acceptodds
Under review as a conference paper at ICLR 2027

Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training

Abstract

A foundational principle in cognitive science holds that intelligent agents do not learn by storing experiences as isolated instances, but by forming abstract schemas that capture relational structure shared across situations. Even though this claim is well supported by behavioral and neuroimaging studies, its role as a computational training signal in language models remains underexplored. We target this gap in the setting of non-stationary language model training, asking whether biasing learning toward structural abstraction reduces catastrophic interference and improves relational generalization as predicted by human results. To study this question, we introduce Abstraction-Augmented Training (AAT), a lightweight loss-level modification that jointly optimizes over concrete instances and their structural abstractions, and two benchmarks, the Relational Cycle Benchmark (RCB) and the Narrative Abstraction Benchmark (NAB). These resources operationalize core cognitive constructs: entity masking as a computational analog of relational alignment, and proverbs as vehicles for implicit abstract meaning that must be inferred across surface-dissimilar situations. Our empirical results demonstrate that AAT improves generalization and retention in a pattern that aligns with cognitive predictions for schema-based learning. Beyond the practical implications for continual learning, these results offer preliminary computational evidence that structural abstraction can serve as a signal for stable learning in non-stationary environments.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Structural abstraction, continual learning, catastrophic forgetting, relational generalization, analogical reasoning, language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_52YcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores structural abstraction as a loss level inductive bias for transferability of learned knowledge. The study is motivated by behavioural and theoretical works from cognitive science. The authors introduce Abstraction-Augmented Training and two benchmark, Relational Cycle Benchmark and Narrative Abstraction Benchmark.

Critical strengths
  • The paper is very well written, and the motivation is strong and clear from the cognitive science literature, connecting behavioural and theoretical work on abstraction as a key property for humans in regards to transferring knowledge across contexts. The authors propose a simple loss-level inductive bias, Abstraction-Augmented Training (AAT) in line with this, by optimising over both the concrete instance and its structural abstraction.
  • The authors contribute two complimentary benchmarks, which are useful for the investigation into the benefits of AAT.
Critical weaknesses
  • Modest Empirical Benefits: (Table 3) AAT achieves the highest unknown accuracy across all three models, but its benefit over alternative methods is modest. Overall cumulative accuracy is similar in comparison to alternative methods, and the benefits in forgetting are mixed. Furthermore, figure 2 shows some benefits in the learning dynamics in comparison to standard SFT, it would also be useful to compare against the other methods in table 3 to establish wether AAT offers similar benefits.
  • Benefit of Abstraction: Table 2 compares structured abstractions with random abstractions. It is unclear whether the structured abstraction adds more benefit then simply presenting the same facts in different words.
Questions for authors
  • Advantages Over Alternative Methods (W1): Given the emricial improvements are modest, what are the practical benefits of the methods in comparison to the methods comparing against (ie. EWC, A-GEM etc.). A comparison of computational cost and other relevant requirements would help clarify when AAT is suitable to use.
  • Paraphrase Control (W2): Would replacing the abstract example with a paraphrase that keeps the original names and facts give similar results with the same training budget?
Reviewer_ysh3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a data augmentation method to explicitly mix/mask out parts of the training data to help the model learn the structure underlying the data better. They propose two variants of the masking (on the categories and entities of the data) and mix this masked-data with the regular data with some alpha ratio. They demonstrate their method on two new benchmarks/datasets that they propose based on Wiki data and proverbs. Based on Figure 2, the forgetting and memorization seem to be slightly better in the small data regime, but as the number of steps / data regime increases, the baseline and AAT seem comparable. Also, it is unclear whether the results are statistically significant, given the multiple seeds of results that they report.

Critical strengths
  • The idea itself is clearly described that they want to improve the models' learning of the structure within the data and they scope out the study in a small, synthetic setting with Wiki and proverb data to do so.
Critical weaknesses
  • Forgetting is only assessed on the relevant dataset that's being trained on. Seems like an insufficient look at how training on this data affects forgetting on past knowledge (which is what we actually care about in continual learning, not just the current dataset we are actively training on)
  • The details on the scoring a bit strange: They look at the raw scores and use a "cutoff" score that's calibrated on some held-out data, but how do they calibrate and why do they use this cutoff-based scoring mechanism instead of generating from the model?
  • How to scale this up for actual SFT or actual pre-training?
  • It also seems that EWC vs AAT seem to be pretty comparalbe on the RCB and NAB tasks across the 3 models (Qwen-1.5B, 3B and SmolLM-1.7B), so not sure when AAT is more beneficial.
Questions for authors
  • Unclear on how they do category vs entity masking (missing some details in the method section). Could the authors clarify and these kinds of details should be in the methods or experimental section clearly?
  • Could you further elaborate why category vs entity is better or worse? Conclusions seem to have drawn from ~0.003-0.005 differences, which don't seem statistically significant
  • Could the authors perhaps explain why is it perhaps that SmolLM-1.7B benefit from SFT or any kind of learning on this dataset, as opposed to the Qwen (1.5B and 3B) models?
Reviewer_kmV7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method to finetune LLMs by abstraction, where the model receives abstract versions of data while learning. The paper gives a simple theoretical result for an idealized setting. Then empirical results using their benchmark datasets are given, showing better performance compared to existing approaches.

Critical strengths

The motivation for using abstraction inspired by cognitive science is interesting. The design of the benchmarks and experiments is reasonably well done.

Critical weaknesses

The method (Algorithm 1) is essentially to augment the dataset with abstracted versions of the original data, such as removal of concrete entity features. This sounds like a simplistic trick rather than a novel technique.

The empirical results (Tables 2, 3) show that the proposed method is mostly the best. However, most of the numbers are so close to the second best that the improvement seems not so significant.

Questions for authors

How do you obtain abstract versions of a passage? Is it just replacing nouns with e1, e2, ... like in Figure 1c?

What's the relationship with studies by Webb et al.?

Comments on the paper

2 comments

I agree with the authors that the complexity of the algorithm should not be the judge of whether it is a good contribution or not and does not have to do with novelty. Given the moderate improvements that AAT brings (EWC slightly beats AAT on overall cumulative accuracy and on Qwen 3B there is no method significantly better than SFT) that AAT brings to the table, it is yet unclear whether this generalizes to real datasets (for example real continual learning setups well-studied in LLM literature [1]) beyond the two benchmarks that the authors propose. The authors explain that one could potentially use pre-trained entity detectors, but I recommend that the authors actually showcase that this is feasible and useful on more realistic datasets. This could really strengthen the paper.

[1] Shenfeld, Idan, et al. "Self-distillation enables continual learning." arXiv preprint arXiv:2601.19897 (2026).

We thank the reviewer for their continued engagement. We recognize that the paper could better explain how our benchmarks build on established continual learning settings.

Continual learning in NLP has been studied under several paradigms, including task-incremental, language-incremental, and online continual knowledge learning (OCKL). OCKL (Jang et al., 2021; Wu et al., 2023) investigates sequential knowledge acquisition from evolving data streams without explicit task/domain boundaries. Prior OCKL studies typically represent knowledge updates as isolated facts or sentences. Our work builds on this established setting by introducing richer, paragraph-level updates involving multiple entities and their relations.

More importantly, we introduce a distinction absent from conventional OCKL evaluations, episodic retention of observed relations versus schematic generalization to logically inferable but unobserved relations.

We appreciate the reviewer's suggestion to investigate other continual learning settings. We have acknowledged these extensions as future directions in our limitations section. We will also expand the background in Section 3 to clarify how our benchmarks relate to established OCKL research and what they contribute beyond it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.