- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces FSMem (Finite-State Machine Memory), a state-transition framework that builds supervised training data from execution trajectories, and it uses this framework to manipulate three properties of the data while the underlying task rules stay fixed. The representational unit is a finite-state machine. Entities, action types, a finite state space and a transition function define a task world, and the transition function encodes the rules that govern when an action is valid and how it changes the state. Trajectories of concrete actions and states become the records from which supervised pairs are built. The authors define action complexity as the number of distinct action types in the corpus, trajectory breadth as the number of distinct state-action pairs covered at each execution step, and trajectory depth as the number of actions in a trajectory. The framework is instantiated in three task families. Synthetic worlds are generated from the state-transition formulation. Classical planning uses problem instances from 13 ACPBench domains. Code execution uses CodeI/O programs, where recorded operations supervise next-operation prediction and outputs supervise program-output prediction in a Native view and an Explicit-trace view. A Qwen2.5-0.5B-Instruct model is fine-tuned with LoRA under a procedure held fixed within each comparison, and rule internalization is measured behaviourally by valid-action identification, execution-outcome prediction, and next-operation prediction in code. Order-robust accuracy is required under every tested candidate ordering.
Critical strengths
Critical strength 1. A manipulable framework for a question that is usually studied with confounded proxies. The paper separates what rules the training data contains from how executions of those rules are recorded, and it does so with a representation that can be instantiated in three unrelated settings (synthetic finite-state worlds, 13 ACPBench planning domains, and CodeI/O programs). Within each setting the transition function stays fixed while action complexity, state-action coverage and trajectory depth are varied on purpose, which is exactly what a study of data organization needs. This matters for my overall assessment because the paper's question is normally addressed with observational proxies in which rule content and data organization move together. The behavioural measurements are also complementary rather than a single score. Valid-action identification probes when an action can be fired, and outcome prediction probes what the action does, so the two measurements map onto the two parts of the transition function that the paper claims must be internalized. Critical strength 2. Findings that replicate across domains and models, and that translate into a cheap practical fix. The breadth effect appears in 12 of 13 planning domains, in all 16 complete model-domain comparisons for the three larger models, and at every candidate count for Blocksworld and Logistics. The cumulative-corpus effect holds in all 19 complete model-domain comparisons, with gains from depth 2 to depth 2+4 to depth 2+4+6 in every comparison. Depth specialization is visible both in the accuracy matrices (Figure 6, Table 1) and in the positive matching gain of all 25 matrices. Because these checks are consistent, the practical recommendation is credible and inexpensive to adopt, namely that a fine-tuning corpus which keeps short trajectories alongside long ones removes most of the depth-specialization penalty at no extra training cost. This is the part of the paper that most supports acceptance.
Critical weaknesses
Critical weakness 1. The amount of supervision per rule is not separated from the organization of the data. All three studies vary one dimension while holding the number of optimizer updates and the number of processed examples fixed, and that choice also changes how much supervision each individual rule receives. In the action-complexity study, expanding the action types from A = 5 to A = 20 while processing the same number of examples means that the five shared action types evaluated afterwards receive far fewer training examples at A = 20 than at A = 5, if queries are spread across the action types that the corpus contains. The reported drop from 88.1% to 65.1% (Lines 257 to 262, Figure 2a) is therefore compatible with a dilution effect in which each rule is simply seen less often, and the result does not by itself establish that unchanged rules become harder to internalize when they are learned alongside more action types. A related coupling appears in the breadth study, where increasing N necessarily reduces repeated use of each source, as the authors note at Line 324, and in the depth study, where conditions that contain only long trajectories consume more tokens at a matched update count. Why it matters. The headline reading of Section 4 concerns interference between rules, and the paper's guidance for data design inherits that reading. If the effect is mostly supervision density, the claim should be restated as a data-efficiency statement, which is still useful but materially weaker. Resolution. A density-matched control would settle it. Hold the number of examples or tokens per action type fixed while adding action types, and report whether accuracy on the shared action types still falls. Critical weakness 2. Scale and ecological validity of the evidence. Every quantitative claim in the main text rests on a single model, Qwen2.5-0.5B-Instruct (Lines 223 to 233). The three larger models in Appendix C cover valid-action identification only, and the authors state that those experiments do not evaluate execution-outcome prediction and do not establish token savings, and that the model columns contain different sets of completed domains and should not be read as a controlled size sweep (Lines 900 to 904). Rule content is also programmatic throughout, since synthetic worlds, planning domains and CodeI/O programs define validity through execution conditions and a transition function, with fixed action identifiers and no natural-language ambiguity. Why it matters. The stated goal is guidance for designing training data for parametric memory, and the title refers to memory formation, whereas the evidence is behavioural and confined to one small model on programmatically defined tasks. A reader may over-read the title as promising mechanistic insight, and the practical recipe may not transfer to corpora that contain instructions, distractors and variable surface forms. Resolution. Repeat one breadth or one depth experiment on a natural-language task family, and report the same protocol at a second model scale including outcome prediction and token accounting.
Questions for authors
Q1 (identification of the action-complexity effect). When the action types grow from A = 5 to A = 20, how does the number of training examples or supervised tokens per action type change? If the corpus were rebalanced so that each action type keeps the same amount of supervision, would the accuracy drop on the five shared action types still appear? This comparison separates interference between rules from a reduction in per-rule supervision, and I would weigh the Section 4 claim differently depending on the answer. Q2 (token accounting in the depth study). The cumulative and long-only conditions are compared at the same number of optimizer updates and examples, and the token counts are reported afterwards. If the comparison were instead made at a matched token budget, so that the long-only or 16-only condition receives more updates for the same number of tokens, would the cumulative corpus still win? The reported token differences in Table 2 are only 2.6% to 5.6%, so the answer changes how the "fewer tokens" claim should be read. Q3 (metric consistency in Table 2). Logistics is evaluated by candidate selection while the other domains in Table 2 are evaluated by exact-set accuracy, and Appendix A then reports exact-set accuracy for all 13 domains. Could the authors recompute the whole of Table 2 under one metric so that the main text and the appendix can be compared directly? Q4 (stability of the code results). The next-operation result in code is described as a single-seed comparison, and the authors note that code token counts are not exactly matched. Do the direction and the size of the effect survive additional seeds, and does the breadth result in the Explicit-trace view remain larger than the Native view? Q5 (reproducibility material). The submission has no Data or Code Availability statement, no hyperparameter table and no prompt templates. Will the corpus construction scripts, the evaluation harness, the LoRA configuration and the query templates be released? In particular, how are candidate orderings generated and sampled for order-robust accuracy, since that sampling rule determines what the metric penalizes?
3 comments
Response to Reviewer mqPT (2)
W2: Model scale and natural-language tasks
Larger model scale. App. C already reports valid-action identification results on Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-8B. To extend this evidence to execution-outcome prediction and token accounting, we added a synthetic depth comparison with Qwen2.5-3B-Instruct, using the same frozen trajectories and fine-tuning protocol as the 0.5B experiment (, with two execution-condition checks per action). Both conditions use 600 optimizer updates and 9,600 processed examples.
| Training depths | Average exact accuracy (%) | Depth-16 accuracy (%) | Non-padding tokens (M) |
|---|---|---|---|
| 1+4+8+16 | 99.79 ± 0.07 | 99.33 ± 0.29 | 3.965 |
| 16-only | 95.33 ± 0.36 | 98.67 ± 0.29 | 6.025 |
Accuracy is mean ± SD across seeds 13, 37, and 73. Average exact accuracy covers the same held-out instances at evaluation depths 1, 4, 8, and 16. Cumulative training achieves higher average exact accuracy in all three seeds, improving the mean by 4.46 percentage points while using 34.2% fewer non-padding training tokens. These results support the benefit of cumulative training at a second model scale in the tested synthetic setting.
Natural-language tasks. Our original tasks use explicit rules so that corpus structure is measurable. As a preliminary check beyond programmatically defined tasks, we repeated the breadth comparison with Qwen2.5-0.5B-Instruct on CLUTRR kinship stories [1], whose text is composed from crowd-written paraphrases, keeping the benchmark's original distractor facts. We held action complexity (16 action types), native proof depth (two rule applications), per-step action exposure, and training budgets fixed. Breadth counts distinct state–action pairs per step, ignoring names while preserving gender.
| Training instances | Trajectory breadth (pairs per step) | Accuracy (%) |
|---|---|---|
| 200 | 177 | 27.0 |
| 400 | 291 | 38.5 |
| 600 | 369 | 39.5 |
| 800 | 426 | 43.5 |
Accuracy is measured on 200 held-out stories that include distractor facts; the table reports one run. Across the runs we performed, the highest-breadth condition consistently outperformed the lowest. We view this as initial evidence that the breadth effect is not specific to programmatically defined tasks.
[1] Sinha, K., et al. CLUTRR: A diagnostic benchmark for inductive reasoning from text. (EMNLP 2019).
Q3: One metric throughout Table 2
We recomputed Table 2 using the full-set results and the corresponding training token counts from App. A. All ten domains now use exact-set accuracy.
| Domain | 2 | 2+4 | 2+4+6 | 6-only | ΔAcc (pp) | ΔTokens (%) |
|---|---|---|---|---|---|---|
| Blocksworld | 31.50 | 62.28 | 77.00 | 35.17 | +41.83 | −2.88 |
| Logistics | 1.39 | 0.56 | 0.67 | 0.67 | 0.00 | −3.67 |
| Depots | 6.72 | 11.44 | 11.83 | 6.39 | +5.44 | −3.51 |
| Ferry | 33.72 | 55.78 | 67.78 | 50.17 | +17.61 | −3.38 |
| Grippers | 9.78 | 24.00 | 31.11 | 22.50 | +8.61 | −4.33 |
| Satellite | 32.67 | 66.00 | 95.89 | 55.33 | +40.56 | −4.87 |
| Grid | 31.06 | 45.22 | 47.11 | 36.22 | +10.89 | −3.18 |
| GoldMiner | 37.56 | 77.28 | 99.44 | 69.06 | +30.39 | −4.35 |
| Swap | 35.44 | 91.83 | 99.67 | 35.89 | +63.78 | −2.63 |
| VisitAll | 46.78 | 79.28 | 80.00 | 75.33 | +4.67 | −5.61 |
| Mean | 26.66 | 51.37 | 61.05 | 38.67 | +22.38 | −3.77 |
Accuracy is exact-set accuracy (%), averaged over test depths 2, 4, and 6 and seeds 13, 37, and 73. Each run uses 300 optimizer updates and 4,800 training example presentations. Both Δ columns compare 2+4+6 with 6-only. ΔTokens uses non-padding training tokens, including EOS. The Mean row averages accuracy across domains and computes ΔTokens from the total token counts across all ten domains.
Cumulative training outperforms 6-only training in nine of these domains and ties in Logistics, where both score 0.67%. Mean accuracy is 61.05% against 38.67%, with 3.77% fewer training tokens across the ten domains.
Response to Reviewer mqPT (3)
Q4: Stability of the code results
Next-operation prediction. Yes. The decline holds in all three seeds, with nearly the same size as the original result. We added two training seeds (37 and 73) for code next-operation prediction at and , alongside the original seed 13. All runs use the same 800 training programs, 600 optimizer updates, and identical 500 held-out execution events evaluated under both candidate orderings.
| Action complexity (A) | Order-robust accuracy (%) | Non-padding training tokens |
|---|---|---|
| 5 | 91.40 ± 2.43 | 7,269,358 |
| 20 | 77.87 ± 1.47 | 7,184,822 |
Accuracy is mean ± SD across seeds 13, 37, and 73. The endpoint decline occurs in every seed, averaging 13.53 ± 1.10 percentage points, compared with 13.6 points in the original single-seed result. Both conditions use 19,200 supervised tokens; total non-padding tokens differ by 1.16%.
Breadth effect across code views. Yes. Across three seeds, the breadth gain remains larger in the Explicit-trace view than in the Native view. We repeated the Figure 5 comparison at all four values of with two additional seeds, holding optimizer updates and example presentations fixed.
| Training programs () | Native accuracy (%) | Explicit-trace accuracy (%) |
|---|---|---|
| 50 | 18.07 ± 1.63 | 18.27 ± 0.31 |
| 200 | 21.00 ± 0.87 | 26.00 ± 1.40 |
| 800 | 26.67 ± 0.50 | 35.27 ± 0.23 |
| 1,600 | 28.13 ± 0.64 | 37.13 ± 0.23 |
Values are mean ± SD across seeds 13, 37, and 73. From to , Explicit-trace gains more in every seed, averaging 18.87 ± 0.42 percentage points versus 10.07 ± 1.89 for Native.
Q5: Reproducibility materials and candidate ordering
Reproducibility materials. Yes. These materials are already available in the supplementary material (https://openreview.net/attachment?id=Pm8NAbVzzq&name=supplementary_material) submitted with the paper, which provides the corpus construction scripts, training and evaluation code, LoRA settings, and prompt generation code. Its README lists the experiment entry points and the required external data. We added a footnote in Sec. 3 directing readers to the Supplementary Material on the OpenReview submission page, and App. B.2 now reports the fine-tuning settings.
Candidate ordering. We fix the candidate set for each test query and evaluate every constructed order deterministically.
- Synthetic valid-action identification and code next-operation prediction use two candidates and test both orders.
- Planning uses C = 2, 3, 4, or 8 candidates. We construct C orders by placing the correct action once at each position while preserving the relative order of the distractors.
Order-robust accuracy is the fraction of queries answered correctly in every tested order. A single error among the tested orders makes the query incorrect.
Response to Reviewer mqPT (1)
Dear Reviewer
mqPT,We sincerely thank you for your careful reading and constructive feedback. We hope the responses below address your concerns and answer your questions.
W1 (action complexity) / Q1: Supervision per action type
We agree that matching the total example budget alone does not control supervision per action type. In short, the answer to Q1 is yes: with per-action supervision fixed, accuracy on the five shared action types still falls from 79.67% to 57.67%. Below, we first report the per-action counts and then the control that holds them fixed.
Per-action counts. In the original Figure 2a experiment, each of the five shared action types is the query target in 160 distinct training episodes at A=5 and 40 at A=20. With two candidate orders and three training epochs, these correspond to 960 and 240 supervised presentations per type. Mean supervised answer tokens, including EOS, decrease from 17,520 to 4,383 per type, averaged over the five shared types and the ten synthetic settings. All counts are per training run.
Control with supervision held fixed. Synthetic next-state prediction (Figure 3) uses the same nested sets of action types as Figure 2a. Each Figure 2a episode contains an executed action and a separate validity query, which can involve different action types. In Figure 3, each training example supervises the resulting state of one executable action, so we can match complete examples and training exposure for each shared action type. At A=10, 15, and 20, each receives the same 80 distinct examples, 480 presentations, and 4,800 supervised tokens at identical training positions. The remaining 400 examples are allocated to auxiliary action types, which requires starting at A=10. All conditions use Qwen2.5-0.5B-Instruct, 800 distinct examples, 300 optimizer updates, and 715,200 non-padding training tokens.
Results are mean ± SD across seeds 13, 37, and 73 on the same 200 held-out queries involving the five shared action types. Accuracy decreases at both steps in every seed. At A=20, rebalancing raises accuracy from 47.67% to 57.67%. In next-state prediction, a 22.0-percentage-point gap to A=10 remains with supervision matched for the five shared action types. Reduced exposure therefore accounts for the smaller part of the decline, and the finding holds with per-action supervision matched.
W1 (trajectory breadth): Coverage and source repetition
At fixed optimizer updates and example presentations, increasing the number of distinct sources expands state–action coverage and reduces repetition per source. We clarified it in Sec. 5.1 that the experiment measures how this allocation of a fixed training budget affects rule reuse.
W1 (trajectory depth) / Q2: Equal token budget
At an equal token budget, cumulative training still outperforms 6-only training.
We matched non-padding training-token budgets on Blocksworld and Grid using Qwen2.5-0.5B-Instruct. We retained the original 300-update 6-only models. Since cumulative training uses fewer tokens per update on average, we trained the 2+4+6 models for 309 updates in Blocksworld and 310 in Grid to match each 6-only budget exactly.
Values are exact-set accuracy (%) averaged over evaluation depths 2, 4, and 6, reported as mean ± SD across seeds 13, 37, and 73 on identical held-out queries. We reevaluated the original 6-only checkpoints using the same inference settings as the token-matched runs. We reevaluated the original 6-only checkpoints using the same inference settings as the token-matched runs, so their values can differ slightly from App. A. At matched token budgets, cumulative training improves average accuracy across the three evaluation depths in every seed for both domains.