acceptodds
Under review as a conference paper at ICLR 2027

Internalized Experience: How Training Data Shapes Memory Formation in LLMs

Abstract

Parametric memory enables large language models (LLMs) to retain information from past experience and reuse it in subsequent tasks. Yet how training data supports the internalization of reusable task rules remains unclear. We introduce FSMem (Finite-State Machine Memory), a state-transition framework for constructing training data and studying how data content and organization shape memory formation. We evaluate rule reuse in synthetic worlds, classical planning, and code execution through two tasks: identifying valid actions and predicting execution outcomes. Our experiments reveal several systematic learning patterns. Under matched training conditions, even unchanged task rules become harder to internalize when learned alongside a larger variety of action types; by contrast, scaling the training data to provide broader state–action coverage generally improves rule learning. We further find that training at a single execution depth produces depth-specialized behavior, with models performing best when training and test executions have similar lengths. Motivated by this finding, we construct cumulative corpora that combine short and long trajectories, which can outperform long-trajectory-only training while using fewer training tokens. Together, these results demonstrate that the same task rules can be internalized differently depending on how execution experience is constructed, and that understanding these learning patterns can guide the design of more effective training data for parametric memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Memory, large language models, training data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mqPTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces FSMem (Finite-State Machine Memory), a state-transition framework that builds supervised training data from execution trajectories, and it uses this framework to manipulate three properties of the data while the underlying task rules stay fixed. The representational unit is a finite-state machine. Entities, action types, a finite state space and a transition function define a task world, and the transition function encodes the rules that govern when an action is valid and how it changes the state. Trajectories of concrete actions and states become the records from which supervised pairs are built. The authors define action complexity as the number of distinct action types in the corpus, trajectory breadth as the number of distinct state-action pairs covered at each execution step, and trajectory depth as the number of actions in a trajectory. The framework is instantiated in three task families. Synthetic worlds are generated from the state-transition formulation. Classical planning uses problem instances from 13 ACPBench domains. Code execution uses CodeI/O programs, where recorded operations supervise next-operation prediction and outputs supervise program-output prediction in a Native view and an Explicit-trace view. A Qwen2.5-0.5B-Instruct model is fine-tuned with LoRA under a procedure held fixed within each comparison, and rule internalization is measured behaviourally by valid-action identification, execution-outcome prediction, and next-operation prediction in code. Order-robust accuracy is required under every tested candidate ordering.

Critical strengths

Critical strength 1. A manipulable framework for a question that is usually studied with confounded proxies. The paper separates what rules the training data contains from how executions of those rules are recorded, and it does so with a representation that can be instantiated in three unrelated settings (synthetic finite-state worlds, 13 ACPBench planning domains, and CodeI/O programs). Within each setting the transition function stays fixed while action complexity, state-action coverage and trajectory depth are varied on purpose, which is exactly what a study of data organization needs. This matters for my overall assessment because the paper's question is normally addressed with observational proxies in which rule content and data organization move together. The behavioural measurements are also complementary rather than a single score. Valid-action identification probes when an action can be fired, and outcome prediction probes what the action does, so the two measurements map onto the two parts of the transition function that the paper claims must be internalized. Critical strength 2. Findings that replicate across domains and models, and that translate into a cheap practical fix. The breadth effect appears in 12 of 13 planning domains, in all 16 complete model-domain comparisons for the three larger models, and at every candidate count for Blocksworld and Logistics. The cumulative-corpus effect holds in all 19 complete model-domain comparisons, with gains from depth 2 to depth 2+4 to depth 2+4+6 in every comparison. Depth specialization is visible both in the accuracy matrices (Figure 6, Table 1) and in the positive matching gain of all 25 matrices. Because these checks are consistent, the practical recommendation is credible and inexpensive to adopt, namely that a fine-tuning corpus which keeps short trajectories alongside long ones removes most of the depth-specialization penalty at no extra training cost. This is the part of the paper that most supports acceptance.

Critical weaknesses

Critical weakness 1. The amount of supervision per rule is not separated from the organization of the data. All three studies vary one dimension while holding the number of optimizer updates and the number of processed examples fixed, and that choice also changes how much supervision each individual rule receives. In the action-complexity study, expanding the action types from A = 5 to A = 20 while processing the same number of examples means that the five shared action types evaluated afterwards receive far fewer training examples at A = 20 than at A = 5, if queries are spread across the action types that the corpus contains. The reported drop from 88.1% to 65.1% (Lines 257 to 262, Figure 2a) is therefore compatible with a dilution effect in which each rule is simply seen less often, and the result does not by itself establish that unchanged rules become harder to internalize when they are learned alongside more action types. A related coupling appears in the breadth study, where increasing N necessarily reduces repeated use of each source, as the authors note at Line 324, and in the depth study, where conditions that contain only long trajectories consume more tokens at a matched update count. Why it matters. The headline reading of Section 4 concerns interference between rules, and the paper's guidance for data design inherits that reading. If the effect is mostly supervision density, the claim should be restated as a data-efficiency statement, which is still useful but materially weaker. Resolution. A density-matched control would settle it. Hold the number of examples or tokens per action type fixed while adding action types, and report whether accuracy on the shared action types still falls. Critical weakness 2. Scale and ecological validity of the evidence. Every quantitative claim in the main text rests on a single model, Qwen2.5-0.5B-Instruct (Lines 223 to 233). The three larger models in Appendix C cover valid-action identification only, and the authors state that those experiments do not evaluate execution-outcome prediction and do not establish token savings, and that the model columns contain different sets of completed domains and should not be read as a controlled size sweep (Lines 900 to 904). Rule content is also programmatic throughout, since synthetic worlds, planning domains and CodeI/O programs define validity through execution conditions and a transition function, with fixed action identifiers and no natural-language ambiguity. Why it matters. The stated goal is guidance for designing training data for parametric memory, and the title refers to memory formation, whereas the evidence is behavioural and confined to one small model on programmatically defined tasks. A reader may over-read the title as promising mechanistic insight, and the practical recipe may not transfer to corpora that contain instructions, distractors and variable surface forms. Resolution. Repeat one breadth or one depth experiment on a natural-language task family, and report the same protocol at a second model scale including outcome prediction and token accounting.

Questions for authors

Q1 (identification of the action-complexity effect). When the action types grow from A = 5 to A = 20, how does the number of training examples or supervised tokens per action type change? If the corpus were rebalanced so that each action type keeps the same amount of supervision, would the accuracy drop on the five shared action types still appear? This comparison separates interference between rules from a reduction in per-rule supervision, and I would weigh the Section 4 claim differently depending on the answer. Q2 (token accounting in the depth study). The cumulative and long-only conditions are compared at the same number of optimizer updates and examples, and the token counts are reported afterwards. If the comparison were instead made at a matched token budget, so that the long-only or 16-only condition receives more updates for the same number of tokens, would the cumulative corpus still win? The reported token differences in Table 2 are only 2.6% to 5.6%, so the answer changes how the "fewer tokens" claim should be read. Q3 (metric consistency in Table 2). Logistics is evaluated by candidate selection while the other domains in Table 2 are evaluated by exact-set accuracy, and Appendix A then reports exact-set accuracy for all 13 domains. Could the authors recompute the whole of Table 2 under one metric so that the main text and the appendix can be compared directly? Q4 (stability of the code results). The next-operation result in code is described as a single-seed comparison, and the authors note that code token counts are not exactly matched. Do the direction and the size of the effect survive additional seeds, and does the breadth result in the Explicit-trace view remain larger than the Native view? Q5 (reproducibility material). The submission has no Data or Code Availability statement, no hyperparameter table and no prompt templates. Will the corpus construction scripts, the evaluation harness, the LoRA configuration and the query templates be released? In particular, how are candidate orderings generated and sampled for order-robust accuracy, since that sampling rule determines what the metric penalizes?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mqPT (1)

Dear Reviewer mqPT,

We sincerely thank you for your careful reading and constructive feedback. We hope the responses below address your concerns and answer your questions.

W1 (action complexity) / Q1: Supervision per action type

We agree that matching the total example budget alone does not control supervision per action type. In short, the answer to Q1 is yes: with per-action supervision fixed, accuracy on the five shared action types still falls from 79.67% to 57.67%. Below, we first report the per-action counts and then the control that holds them fixed.

Per-action counts. In the original Figure 2a experiment, each of the five shared action types is the query target in 160 distinct training episodes at A=5A=5 and 40 at A=20A=20. With two candidate orders and three training epochs, these correspond to 960 and 240 supervised presentations per type. Mean supervised answer tokens, including EOS, decrease from 17,520 to 4,383 per type, averaged over the five shared types and the ten synthetic settings. All counts are per training run.

Control with supervision held fixed. Synthetic next-state prediction (Figure 3) uses the same nested sets of action types as Figure 2a. Each Figure 2a episode contains an executed action and a separate validity query, which can involve different action types. In Figure 3, each training example supervises the resulting state of one executable action, so we can match complete examples and training exposure for each shared action type. At A=10A=10, 15, and 20, each receives the same 80 distinct examples, 480 presentations, and 4,800 supervised tokens at identical training positions. The remaining 400 examples are allocated to auxiliary action types, which requires starting at A=10A=10. All conditions use Qwen2.5-0.5B-Instruct, 800 distinct examples, 300 optimizer updates, and 715,200 non-padding training tokens.

Action complexity (A)Exact next-state accuracy (%)
1079.67 ± 2.36
1562.50 ± 5.63
2057.67 ± 4.16

Results are mean ± SD across seeds 13, 37, and 73 on the same 200 held-out queries involving the five shared action types. Accuracy decreases at both steps in every seed. At A=20A=20, rebalancing raises accuracy from 47.67% to 57.67%. In next-state prediction, a 22.0-percentage-point gap to A=10A=10 remains with supervision matched for the five shared action types. Reduced exposure therefore accounts for the smaller part of the decline, and the finding holds with per-action supervision matched.

W1 (trajectory breadth): Coverage and source repetition

At fixed optimizer updates and example presentations, increasing the number of distinct sources expands state–action coverage and reduces repetition per source. We clarified it in Sec. 5.1 that the experiment measures how this allocation of a fixed training budget affects rule reuse.

W1 (trajectory depth) / Q2: Equal token budget

At an equal token budget, cumulative training still outperforms 6-only training.

We matched non-padding training-token budgets on Blocksworld and Grid using Qwen2.5-0.5B-Instruct. We retained the original 300-update 6-only models. Since cumulative training uses fewer tokens per update on average, we trained the 2+4+6 models for 309 updates in Blocksworld and 310 in Grid to match each 6-only budget exactly.

DomainTokens per condition6-onlyCumulative (2+4+6)
Blocksworld1,466,37635.17 ± 0.7377.11 ± 1.06
Grid4,039,33236.06 ± 2.1847.44 ± 0.35

Values are exact-set accuracy (%) averaged over evaluation depths 2, 4, and 6, reported as mean ± SD across seeds 13, 37, and 73 on identical held-out queries. We reevaluated the original 6-only checkpoints using the same inference settings as the token-matched runs. We reevaluated the original 6-only checkpoints using the same inference settings as the token-matched runs, so their values can differ slightly from App. A. At matched token budgets, cumulative training improves average accuracy across the three evaluation depths in every seed for both domains.

AuthorsOpenReview ↗

Response to Reviewer mqPT (2)

W2: Model scale and natural-language tasks

Larger model scale. App. C already reports valid-action identification results on Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-8B. To extend this evidence to execution-outcome prediction and token accounting, we added a synthetic depth comparison with Qwen2.5-3B-Instruct, using the same frozen trajectories and fine-tuning protocol as the 0.5B experiment (A=10A=10, with two execution-condition checks per action). Both conditions use 600 optimizer updates and 9,600 processed examples.

Training depthsAverage exact accuracy (%)Depth-16 accuracy (%)Non-padding tokens (M)
1+4+8+1699.79 ± 0.0799.33 ± 0.293.965
16-only95.33 ± 0.3698.67 ± 0.296.025

Accuracy is mean ± SD across seeds 13, 37, and 73. Average exact accuracy covers the same held-out instances at evaluation depths 1, 4, 8, and 16. Cumulative training achieves higher average exact accuracy in all three seeds, improving the mean by 4.46 percentage points while using 34.2% fewer non-padding training tokens. These results support the benefit of cumulative training at a second model scale in the tested synthetic setting.

Natural-language tasks. Our original tasks use explicit rules so that corpus structure is measurable. As a preliminary check beyond programmatically defined tasks, we repeated the breadth comparison with Qwen2.5-0.5B-Instruct on CLUTRR kinship stories [1], whose text is composed from crowd-written paraphrases, keeping the benchmark's original distractor facts. We held action complexity (16 action types), native proof depth (two rule applications), per-step action exposure, and training budgets fixed. Breadth counts distinct state–action pairs per step, ignoring names while preserving gender.

Training instancesTrajectory breadth (pairs per step)Accuracy (%)
20017727.0
40029138.5
60036939.5
80042643.5

Accuracy is measured on 200 held-out stories that include distractor facts; the table reports one run. Across the runs we performed, the highest-breadth condition consistently outperformed the lowest. We view this as initial evidence that the breadth effect is not specific to programmatically defined tasks.

[1] Sinha, K., et al. CLUTRR: A diagnostic benchmark for inductive reasoning from text. (EMNLP 2019).

Q3: One metric throughout Table 2

We recomputed Table 2 using the full-set results and the corresponding training token counts from App. A. All ten domains now use exact-set accuracy.

Domain22+42+4+66-onlyΔAcc (pp)ΔTokens (%)
Blocksworld31.5062.2877.0035.17+41.83−2.88
Logistics1.390.560.670.670.00−3.67
Depots6.7211.4411.836.39+5.44−3.51
Ferry33.7255.7867.7850.17+17.61−3.38
Grippers9.7824.0031.1122.50+8.61−4.33
Satellite32.6766.0095.8955.33+40.56−4.87
Grid31.0645.2247.1136.22+10.89−3.18
GoldMiner37.5677.2899.4469.06+30.39−4.35
Swap35.4491.8399.6735.89+63.78−2.63
VisitAll46.7879.2880.0075.33+4.67−5.61
Mean26.6651.3761.0538.67+22.38−3.77

Accuracy is exact-set accuracy (%), averaged over test depths 2, 4, and 6 and seeds 13, 37, and 73. Each run uses 300 optimizer updates and 4,800 training example presentations. Both Δ columns compare 2+4+6 with 6-only. ΔTokens uses non-padding training tokens, including EOS. The Mean row averages accuracy across domains and computes ΔTokens from the total token counts across all ten domains.

Cumulative training outperforms 6-only training in nine of these domains and ties in Logistics, where both score 0.67%. Mean accuracy is 61.05% against 38.67%, with 3.77% fewer training tokens across the ten domains.

AuthorsOpenReview ↗

Response to Reviewer mqPT (3)

Q4: Stability of the code results

Next-operation prediction. Yes. The decline holds in all three seeds, with nearly the same size as the original result. We added two training seeds (37 and 73) for code next-operation prediction at A=5A=5 and A=20A=20, alongside the original seed 13. All runs use the same 800 training programs, 600 optimizer updates, and identical 500 held-out execution events evaluated under both candidate orderings.

Action complexity (A)Order-robust accuracy (%)Non-padding training tokens
591.40 ± 2.437,269,358
2077.87 ± 1.477,184,822

Accuracy is mean ± SD across seeds 13, 37, and 73. The endpoint decline occurs in every seed, averaging 13.53 ± 1.10 percentage points, compared with 13.6 points in the original single-seed result. Both conditions use 19,200 supervised tokens; total non-padding tokens differ by 1.16%.

Breadth effect across code views. Yes. Across three seeds, the breadth gain remains larger in the Explicit-trace view than in the Native view. We repeated the Figure 5 comparison at all four values of NN with two additional seeds, holding optimizer updates and example presentations fixed.

Training programs (NN)Native accuracy (%)Explicit-trace accuracy (%)
5018.07 ± 1.6318.27 ± 0.31
20021.00 ± 0.8726.00 ± 1.40
80026.67 ± 0.5035.27 ± 0.23
1,60028.13 ± 0.6437.13 ± 0.23

Values are mean ± SD across seeds 13, 37, and 73. From N=50N=50 to N=1,600N=1{,}600, Explicit-trace gains more in every seed, averaging 18.87 ± 0.42 percentage points versus 10.07 ± 1.89 for Native.

Q5: Reproducibility materials and candidate ordering

Reproducibility materials. Yes. These materials are already available in the supplementary material (https://openreview.net/attachment?id=Pm8NAbVzzq&name=supplementary_material) submitted with the paper, which provides the corpus construction scripts, training and evaluation code, LoRA settings, and prompt generation code. Its README lists the experiment entry points and the required external data. We added a footnote in Sec. 3 directing readers to the Supplementary Material on the OpenReview submission page, and App. B.2 now reports the fine-tuning settings.

Candidate ordering. We fix the candidate set for each test query and evaluate every constructed order deterministically.

  • Synthetic valid-action identification and code next-operation prediction use two candidates and test both orders.
  • Planning uses C = 2, 3, 4, or 8 candidates. We construct C orders by placing the correct action once at each position while preserving the relative order of the distractors.

Order-robust accuracy is the fraction of queries answered correctly in every tested order. A single error among the tested orders makes the query incorrect.

Reviewer_P6aXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper studies how the content and organisation of fine-tuning data affect whether LLMs internalise reusable task rules.
  2. They introduce “FSMem”, a framework for constructing training data as well as studying how data content and organization shape memory. The framework varies three things while keeping existing action rules unchanged: the number of action types, breadth (state–action coverage), and trajectory depth.
  3. They test this on synthetic worlds, 13 ACPBench planning domains, and CodeI/O code execution, primarily with Qwen2.5-0.5B + LoRA.
  4. They find that more action types at a fixed budget generally lowers accuracy (even on shared rules) and more distinct sources at a fixed step count generally improves accuracy and single-depth training produces depth-specialised behaviour.
  5. They also find that cumulative corpora mixing short and long trajectories can beat long-only training while using fewer training tokens.
  6. They provide some additional results using Qwen2.5-3B/7B and Qwen3-8B in the appendix.
Critical strengths
  1. Good framework + methodology a. The setup lets you systematically vary action types, breadth and depth while preserving existing action rules. I like this idea and think it’s a nice way to study data construction. b. Their evaluation methodology is pretty sound (held-out sources, repository-disjoint code splits, the same test queries across conditions and order-robust accuracy to avoid candidate-position effects) which makes me feel confident enough

  2. The mixed-depth results are interesting and seem well supported a. Uplift demonstrated across several domains, with additional results with larger models. Gives a practical takeaway for constructing training data. b. Results seem to extend; improvements at the deepest planning depth in 16/19 comparisons (App C.4), and the long executions code experiments, though in the synthetic setting most of the gain comes from covering shorter test depths.

Critical weaknesses
  1. Some effects might just be simpler training-data effects? a. It’s unclear to me how much of the decline comes from learning alongside more rules versus less supervision per rule( e.g if you have a fixed example budget, adding action types may mean the model sees each shared action less often). b. Testing on the same questions may control test difficulty but not training exposure and I couldn’t find per-action counts or a control holding these fixed. c. If the drop does in fact mainly come from less supervision per rule, it doesn;t provide that much evidence that learning additional rules makes the shared rules harder to learn beyond that reduction in exposure.
Questions for authors

I couldn't find these details;

  1. Synthetic action rules, what varies across the 40 configurations, or how actions and candidate distractors are sampled etc
  2. Fine-tuning settings - LoRA rank / target modules and learning rate etc Could you provide these?
1 comment
AuthorsOpenReview ↗

Response to Reviewer P6aX

Dear Reviewer P6aX,

We sincerely thank you for your careful reading and constructive feedback. We hope the responses below address your concerns and answer your questions.

W1: Supervision per rule versus number of rules

We agree that using the same test questions controls test difficulty but not training exposure, and that the paper did not report per-action counts. In short, the decline persists when per-action supervision is held fixed: accuracy on the five shared action types still falls from 79.67% to 57.67%. We report the per-action counts and the control below.

Per-action counts. In the original synthetic action-identification experiment (Figure 2a), each of the five shared action types is the query target in 160 distinct training episodes at A=5A=5 and 40 at A=20A=20. With two candidate orders and three epochs, this corresponds to 960 and 240 supervised presentations per action type.

Control with supervision held fixed. We added a control on synthetic next-state prediction (Figure 3), which uses the same nested sets of action types as Figure 2a. Each training example supervises the resulting state of one executable action, so we can match complete examples and training exposure for each shared action type. At A=10A=10, 15, and 20, each receives the same 80 distinct examples, 480 presentations, and 4,800 supervised tokens at identical training positions. The remaining 400 examples are allocated across 5, 10, or 15 auxiliary action types. All conditions use Qwen2.5-0.5B-Instruct, 800 distinct examples, 300 optimizer updates, and 715,200 non-padding training tokens.

Action complexity (A)Exact next-state accuracy (%)
1079.67 ± 2.36
1562.50 ± 5.63
2057.67 ± 4.16

Results are mean ± SD across seeds 13, 37, and 73 on the same 200 held-out queries involving the five shared action types. Accuracy decreases at both steps in all three seeds.

At A=20A=20, rebalancing supervision raises exact next-state accuracy from 47.67% to 57.67%. A 22.0-percentage-point gap to A=10A=10 remains when supervision is matched for the five evaluated action types. Reduced exposure therefore accounts for the smaller part of the decline, and the finding holds with per-action supervision matched.

Q1: Synthetic rules, configurations, and sampling

  1. Synthetic rules. The action-identification world has two people, two objects, and four locations. The state records entity locations, and each action schema specifies typed arguments, preconditions, and deterministic updates. Let KK denote the number of execution-condition checks that are active. At K=2K=2, move_person(p,x,y) requires pp to be at xx and xx to be connected to yy, then moves pp to yy. The next-state control uses eight locations.
  2. Configurations. We cross A∈{5,10,15,20}A\in\{5,10,15,20\} with K∈{2,3,4,5,6,8,10,12,14,16}K\in\{2,3,4,5,6,8,10,12,14,16\}, which gives the 40 configurations. At fixed KK, larger AA adds schemas while preserving existing rules. Each KK activates the first KK checks in order, starting with connectivity and source presence, then destination capacity, entity compatibility, edge permission, departure and entry clearance, origin and destination permission, role-class authorization, class-route clearance, moving-group compatibility, edge-load and mover capacity, class-distance budget, and action-time budget.
  3. Sampling. We enumerate executable one-step transitions from all 256 states and type-correct groundings. Per configuration, we select 800 training and 200 held-out episodes, balancing executed and queried action types. Each query pairs two groundings of the same schema, one executable and one invalid in the resulting state. We include both candidate orders.

Q2: Fine-tuning settings

We fine-tuned Qwen2.5-0.5B-Instruct with LoRA using the following settings.

SettingValue
OptimizerAdamW
LoRA rank / alpha / dropout8 / 16 / 0
LoRA target modulesAttention query and value projections (q_proj, v_proj)
Peak learning rate2×10−42 \times 10^{-4}
Example presentations per optimizer update16

We added this table to App.B.2 of the revised paper.

Reviewer_d4ECOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the parametric memory formation problem in LLMs. A new state-transition framework called FS-Mem is proposed, for constructing training data and studying how data content and organization shape memory formation. The authors instantiate FSMem in synthetic worlds, classical planning, and code execution. The study controls action complexity, trajectory breadth and depth. Extensive experiments show the competitive performance of proposed method.

Critical strengths
  • The task and problem presented in the paper are important and interesting.
  • The logic is fluent, and overall the paper is well-written.
  • The experiments are extensive, covering different dimensions and aspects.
  • The controlled study of action complexity, trajectory breadth and depth, are convincing.
Critical weaknesses
  • Overall speaking, the core findings in this paper are mostly within expectation, nothing too much surprising. For example, "Expanding trajectory breadth generally improves memory formation", or "Training on a cumulative corpus of short and long trajectories can outperform long-only one". Can the authors summarize a few more interesting or even surprising findings?

  • How is the memory formation related to model transferability or generalizability? This paper mainly claims the concept of "parametric memory formation", but it actually fine-tuned a model, learned the rules of a state-transition system, and applied them to new instances or samples. Throughout the paper, very few literatures in model transferability or generalizability are mentioned or discussed.

  • Because the cumulative corpus includes short trajectories, it will naturally use fewer tokens than the long-only corpus when the number of training examples is the same. Therefore, the token savings are a byproduct of including shorter trajectories, not a genuine efficiency gain.

Questions for authors

Please see above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer d4EC

Dear Reviewer d4EC,

We sincerely thank you for your careful reading and constructive feedback. We hope the responses below address your concerns and answer your questions.

W1: Novelty of the findings

To our knowledge, FSMem is the first state-transition framework to jointly study action complexity, trajectory breadth, and trajectory depth. It enables controlled comparisons while preserving existing task rules.

We agree that some findings could be expected: broader trajectory breadth improves memory formation, and mixing short and long trajectories can outperform long-only training. These are two of the intuitions that our framework can verify under controlled conditions. In this framework, action complexity measures how difficult a corpus is to learn. Breadth and depth are then the two dimensions along which execution records can be organized to cope with that difficulty. Our study also unveils the following insightful findings:

  • Coverage over repetition. The breadth gain does not come from more training. With the number of training examples fixed, covering more state–action pairs means revisiting each source less often, and rule learning still generally improves.
  • Depth specialization. Longer training trajectories do not automatically support stronger performance at shorter depths. Single-depth training produces depth-specialized behavior.
  • Short trajectories help long executions. A cumulative corpus can outperform long-only training even when evaluated only on long executions.

W2: Relationship to generalizability and transferability

In short, generalization is how we measure memory formation, and transfer in our setting is the reuse of a task rule across entities and states within a task. We explain both below, and we added the discussion to the Related Work section.

Memory formation and generalization. Parametric memory formation is what we study: which task rules a model retains in its parameters from execution experience, and how the content and organization of that experience affect this retention. Generalization is how we observe it. Factual memory can be tested by recalling the stored item. A task rule applies across entities and states, so reproducing training records cannot show that the rule was retained. We therefore test on held-out instances governed by the same rules, with the training records absent from the context. Within-task generalization is our behavioral evidence that the model has internalized the rule and not the particular executions. Our depth study is the closest point of contact with the generalization literature: depth specialization is a failure of length generalization, and cumulative corpora are a data-side way to reduce it.

Memory formation and transferability. Transfer learning studies whether knowledge acquired in a source task or domain helps in a target one. In FSMem, what transfers is the task rule. An action type is defined by its execution conditions and effects in the transition function, independently of the entities it is applied to. A rule learned from executions on some entities and states must therefore be reused on other entities and states, and, in the depth study, at other execution depths. This is transfer within a task: training and test share the transition function and differ in the instances it is applied to. It describes the same evaluation as the within-task generalization above, seen from the other side: generalization is the measurement, and the rule is what transfers. We do not study transfer across tasks, where the transition function itself changes.

W3: Interpretation of the token savings

We agree that including shorter trajectories naturally reduces token consumption when the number of training examples is fixed, so the token reduction itself is a byproduct of the corpus composition. Our finding concerns accuracy: replacing some long trajectories with shorter ones improves accuracy, and the improvement is not paid for with more tokens. With the same number of training examples and optimizer updates, training on a cumulative corpus of depths 2+4+6 raises average exact-set accuracy across ten planning domains from 38.67% (6-only) to 61.05%, averaged over test depths 2, 4, and 6, while using 2.6% to 5.6% fewer non-padding training tokens in each domain. These numbers come from Table 2.

Thanks for the detailed response. My main concerns regarding the novelty of the findings, the relationship to generalization and transferability, and the interpretation of the token savings have been adequately addressed. Given the important and interesting problem, the clearly written paper, and the extensive controlled experiments on action complexity, trajectory breadth and depth, I therefore maintain my positive assessment (3: Weak acceptance).

I do not raise my score further, for the following reasons, which concern the overall scope of the work rather than missing experiments:

  1. Limited conceptual novelty. Most findings confirm intuitions under controlled conditions rather than revealing new phenomena, so the main contribution lies in the framework and the systematic verification.
  2. Framing of memory formation. As your response clarifies, memory formation is measured as within-task generalization after fine-tuning. The paper could position itself more clearly relative to the existing literature on generalization and data composition.
  3. Scope of the evidence. Most experiments use small models and structured state-transition tasks. It remains unclear how far the conclusions extend to larger models and to less structured, natural-language settings.
AuthorsOpenReview ↗

Thank you for your thoughtful follow-up

Dear Reviewer d4EC,

Thank you for following up and for maintaining your positive score. We are glad that our response addressed your main concerns, and we appreciate your recognition of the problem, the clarity of the paper, and the controlled experiments.

Following your suggestions, we will clarify the paper’s main contribution as a framework for systematically studying how training data shapes rule reuse. We will also connect our evaluation of within-task generalization more directly to prior work on generalization and data composition.

On the scope of the evidence, App. C reports similar gains from broader trajectory coverage and cumulative depth training in valid-action identification on larger models. In our response to mqPT (W2), we also report that with Qwen2.5-3B-Instruct, training on a cumulative corpus improves average synthetic execution-outcome accuracy by 4.46 pp while using 34.2% fewer training tokens. On CLUTRR kinship stories with crowd-written paraphrases and distractor facts, increasing trajectory breadth raises held-out accuracy from 27.0% to 43.5% in the reported run. We will clarify the scope of these findings in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.