acceptodds
Under review as a conference paper at ICLR 2027

Mimir: A Sub-2B Hierarchical Reasoning Model Trained on Permissible Data Punches Above Its Weight Class

Abstract

Training capable language models typically relies on web-scale pretraining corpora followed by multi-stage post-training, creating substantial computational, as well as data-governance barriers for settings in which training data must satisfy explicit permissibility constraints. We introduce Mimir, a 1.78B-parameter language model based on the Hierarchical Reasoning Model Text architecture, a variant of recurrent transformers where the layer stack is split into L and H stacks that are iterated and interleaved. Mimir is trained from random initialization directly on a curated mixture of instruction, reasoning, tool-use, multilingual, and synthetic data. The training mixture comprises 161 datasets and 70.5B sampled tokens per epoch for Mimir v1, with substantial coverage of both English and Danish and with synthetic replacements for data sources that do not satisfy the target permissibility criteria. Across 20 benchmarks, Mimir substantially outperforms all sub-2B reference models, demonstrating that our permissible data approach produces state-of-the-art results for both high-resource languages (English, 1B speakers) and low-resource languages (Danish, 6M speakers) and is competitive on specialized domains such as mathematical reasoning and code. We conduct two memorisation audits, showing that memorisation risk is generally low even though the model is trained for multiple epochs. We also perform inference time ablations on recurrent depth, demonstrating graceful degradation for recurrence changes of the L stack. Our results provide evidence that hierarchical recurrent language models can learn effectively from post-training-oriented, permissibility-constrained data mixtures without relying on conventional web-scale pretraining. Mimir outclasses the original HRM-Text 1B, as well as the looped transformer Ouro, and represents the new state of the art among frontier sub-2B models, and competes with larger frontier models such as Gemma E4B and Qwen 3.5 9B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language modeling, looped transformers, hierarchical reasoning models, synthetic instruction data

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZwpoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Mimir v1/v1.5 are sub-2B language models built on the HRM-Text architecture, which alternates low-level (L) and high-level (H) modules in a hierarchical recurrent depth. The main contribution is a generate-and-audit pipeline producing the Mimir Training Corpus, on which both models are trained from scratch using only permissible data. The authors report competitive English, Math & Code, and Danish results against frontier models up to 9B, plus two memorisation audits and inference-time recurrence ablations.

Critical strengths

Training competitive small models on legally permissible data remains under-explored, especially for low-resource languages. The generate-and-audit design is concrete, principled, and backed by row-level evidence. The models match or surpass much larger frontier baselines, demonstrating that competitive performance is achievable without web-scale or non-permissible data.

Critical weaknesses

1.HRM-Text was trained on a different data mixture, and the paper does not control for token count, compute, or data composition before claiming that Mimir outperforms HRM-Text 1B on every benchmark except MATH. A fairer comparison would include an HRM-Text-style baseline retrained on the Mimir corpus, or at least a compute-matched discussion.

2.Training compute and token efficiency claims are under-supported. The paper states that v1.5 reaches competitive performance using roughly 10% of Ouro's 7.7T tokens, but Ouro is a looped transformer rather than an HRM; the more relevant comparison, HRM-Text's training token count, is not discussed. Mimir v1 reports about 8,000 B200 hours, but v1.5's compute is not reported.

3.The ablation reveals architectural brittleness. Table 5 shows that changing H from 2 to 1 drops multiple-choice performance below chance, and H=3 fails to produce a parseable answer on 99.3% of GSM8K problems. This indicates that the model is highly specialised to its inference-time schedule, and the practical elasticity of recurrent depth is limited, weakening its deployment value. The paper should discuss whether this is a general property of HRM-Text or specific to Mimir, and whether regularisation could improve it.

Questions for authors

1.What are HRM-Text's training tokens and compute, and can you provide a compute-matched comparison?

2.What is v1.5's total training compute, and why compare token efficiency to Ouro rather than HRM-Text?

3.Is the H-module brittleness specific to Mimir or general to HRM-Text, and could regularisation improve inference-time elasticity?

2 comments

We thank the reviewer for their time spent assesing our work. We thank the reviewer for acknowledging this is an under-explored challenging area. Further, that our generatate-and-audit design is principled and backed by row-level evidence, demonstrating that we can achieve frontier performance without web-scale and non-persmissible data.

HRM-Text comparison.

We thank the reviewer and will clarify that Mimir v1 is itself trained from random initialization using the HRM-Text architecture on the Mimir corpus. Thus, it serves as the requested HRM-Text-style model trained on our data. We agree, however, that its comparison with the original HRM-Text checkpoint is not controlled for training tokens, compute, or all other training choices; the performance difference therefore cannot be attributed to the corpus alone. We will report HRM-Text’s training-token count and compute where available. Mimir’s corpus sizes are 70.5B and 103B tokens per epoch for v1 and v1.5, respectively; these are not total training-token counts.

Compute and Ouro.

We compare with Ouro as another recurrent-depth model in a similar parameter range, not as a controlled HRM-Text comparison. While Mimir uses fewer training tokens than the 7.7T reported for Ouro, token count alone does not establish compute efficiency. We will report v1.5’s total training tokens and GPU-hours, including v1 training [insert verified figures], and qualify the comparison accordingly.

Inference-time brittleness.

We agree that changing the H-cycle count substantially limits the practical elasticity of this checkpoint, particularly for free generation. Our experiment changes the inference schedule of a single Mimir v1 checkpoint, so it cannot establish whether this behaviour is general to HRM-Text. Training with varied recurrence schedules or other regularisation may improve elasticity, but this remains untested future work.

Response to Questions For Authors:

  1. Mimir v1 is an HRM-Text-style model trained from scratch on the Mimir corpus. The comparison with the original HRM-Text checkpoint is not compute- or token-matched, however, and does not isolate the effect of the corpus. HRM-Text’s total training tokens and compute: [insert verified figures, if available].
  2. As we already are training and HRM-Text architecture that makes less sense. We compare to Ouro, as it is the newest looped architecture around Mimir's weight-class. It is trained on a lot more tokens, using alot more compute, but without gaining any superior performance to Mimir.
  3. We believe it is a natural behaviour that a models performs best with the architecture it is trained for, and it is not expected that post-hoc changing of the architecture and hence the weight+activation computation would work. We believe this can work, but will require substanstial more work.

Considering the limitations of this paper, I will maintain my original score.

Reviewer_9VATOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mimir, a 1.78B-parameter model based on the HRM-Text architecture, trained on permissible data to develop English and Danish language capabilities. The authors present two versions, v1 and v1.5, with v1.5 obtained through additional post-training of v1. They report strong performance across multiple benchmarks.

Critical strengths

The paper demonstrates substantial engineering effort. The authors train a small language model from scratch and achieve competitive performance.

Critical weaknesses

The paper reads more like a technical report, detailing how datasets are cleaned to satisfy permissibility requirements and devoting considerable space to the model architecture. Although this is appropriate for a technical report, the contribution appears overly focused on engineering for a research paper.The data curation process appears labor-intensive and requires different procedures for different datasets, but the individual processing techniques seem fairly standard. The paper does not explain what is distinctive about these procedures or how the process can be made more efficient. For example, lines 132–138 discuss potentially erroneous examples in an instruction dataset and report how many were ultimately replaced. A research paper should place greater emphasis on the methodological innovations underlying this process.Similarly, the model architecture is adopted from prior work, yet substantial space is devoted to describing it and the training hyperparameters. These details are useful for understanding and reproducing the model, but their relevance to the paper’s research contribution is unclear.To strengthen its case for acceptance at a research conference, the paper should focus on its distinctive contributions. If the primary contribution is high-quality training data curation, the authors should demonstrate improvements in curation efficiency or validate the benefits of the resulting data across multiple model architectures, rather than simply showing that the final model performs well. From the perspective of a research conference, I believe the paper has substantial shortcomings in its current form.

Questions for authors

What are the paper’s distinctive methodological contributions? Do they improve training efficiency, model capabilities, or both? Can controlled experiments demonstrate that the reported gains arise from these specific contributions?

1 comment

We thank the reviewer and agree that the paper should distinguish its contribution more clearly from prior architecture work. We do not propose a new architecture or claim that each data-processing operation is novel in isolation. Our central research question is how capable a small bilingual model can be when trained from scratch under explicit data-permissibility constraints. Mimir demonstrates one achievable performance level under these constraints, without claiming to establish the limit of such approaches. We contribute an end-to-end, auditable approach to building the corpus, combining generation and auditing with source-grounded text transformations, repair of flawed examples, and conversion of heterogeneous data into bilingual and native tool-use supervision. Sharing training recipes and resources is itself valuable for research, as illustrated by OLMo with Dolma. Here, also fully reproduceable.

Training method and audits. For Mimir v1.5, we divide each recurrent module's learning rate by the number of its applications receiving gradients. We found this scaling critical to stabilising training when extending backpropagation through all eight recurrent steps, as well as when training larger versions, although we have not isolated its effect in a controlled ablation. Two independent memorisation audits of Mimir v1 found little verbatim reproduction under the tested scenarios; these findings do not establish legal compliance or rule out memorisation in other settings.

Transformer comparison. We trained a standard Transformer and a corresponding HRM with matched parameter counts, tokenizer, training-data mixture, token budget, and training compute. At step 393,997 (the end of the first epoch in this comparison), the mean benchmark scores are:

Benchmark groupStandard TransformerMatched HRM (PrefixLM/target-only)HRM difference
Overall31.7140.06+8.35
Danish39.7151.37+11.66
English43.6252.45+8.82
Math & Code34.9741.86+6.90

The HRM scores 6.9–11.7 points higher across the four reported averages. This supports the overall HRM/PrefixLM/target-only recipe, but does not isolate recurrence from differences in attention and loss, or establish the effect of individual curation choices. We do not claim a measured improvement in curation or training efficiency. We will make these limits explicit and foreground the research question and evidence in the paper. We are running further ablations on the exact architectural mechanisms that contribute to the gain. We can confidently report already now that the effects of the different architectural aspects compound.

Reviewer_9DXzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work trained a recurrent depth model from scratch based on HRM-text model architecture on "permissible" only data consisting mixture of instruction, reasoning, tool-use and synthetic data, both for English and Danish languages. The final 1.78B model achieves competitive performance in English/danish general and math/code benchmarks.

Critical strengths
  • The paper provide a curated training corpus and SDG pipelines under data permissibility constraints.
  • From their experimental results in Table 2/3 , the model achieves good performance than baseline in benchmarks GSM8k/HumanEval etc.
Critical weaknesses
  • Limited Novelty: The paper uses the same architecture as earlier works of HRM-Text and SDG pipelines just extend already well established methods from previous works, I didn't find any novelty in the paper.
  • Paper lacks a very important baseline of training standard transformer on the same baseline and data to establish whether the architectural choice is responsible for gains or the datasets, I didn't find any good reason for choosing a Recurrent-depth model (with using Prefix LM attention mask) for this study, given the goal is to work with Permissible Data.
Questions for authors
  • Matched on same compute and inference budget, what's the advantage of using this recurrent depth architecture over standard transformer baseline.
  • Current training mixes several things like recurrent-depth model, SDG datasets and continual training, ablations related to the isolations will be helpful.
  • As noted in limitation section by authors, the from scratch training with reasoning data does not replaces the need of broad knowledge gains from the next token prediction cross entropy loss, its not clear to me why would we want to do from scratch training on instruction/reasoning datasets directly from reading this paper.
3 comments

Novelty

  1. Based on our findings from Mimir v1, we have improved the training pipeline by introducing the learning rate dividers described in Section 3 ("Continued Training: Mimir v1.5"). This is a vital modification to stabilize the training of the HRM architecture. This not only results in stronger performance of the trained models (particularly v1.5) but also is crucial for training stability at larger sizes (e.g. the XXL size defined by the HRM-Text work, which either trains unstably or unacceptably slow without this adaptation).

  2. We would also like to emphasize that the HRM-Text was mainly evaluated on MCQ and single asnwer QA benchmarks and is not very good in generating coherent text. Its performance on mathematics task is judged primarily by the final answer. In contrast, we show that Mimir is capable of free-text generation as evaluated on multiple benchmarks. In particular, Mimir can, among others, translate between languages with coherent language generation, summarize in multiple languages, rewrite wrongly ordered paragraphs, and correct sentences with spelling and/or grammar mistakes For this, we devised and implemented a novel pre-training to post-training data conversion.

  3. While our work builds upon the primary architecture of HRM-Text, we extend the setup by integrating a modern tokenizer—specifically the Gemma 4 tokenizer with chat-template introducing the capability of multi-turn conversations. Consequently, the Mimir family is trained to support both interaction with the user and a modern range of languages. The former, the interactive use, required a new training regime where individual turns are handled correctly regarding the PrefixLM target-only loss-masking

  4. We demonstrate the the HRM-Text Architecture can be trained to a competitive performance on permissible-only data. This is a contribution to open foundation models.

Matched Transformer Comparison

We agree that a standard Transformer baseline is essential for a fair comparison. We have trained a standard Transformer and a corresponding HRM with matched parameter counts, tokenizers, training-data mixtures, token budgets, and total training compute. At step 393,997 (the end of the first epoch in this comparison), the mean benchmark scores are as follows:

Benchmark groupStandard TransformerMatched HRM (PrefixLM/target-only)HRM difference
Overall31.7140.06+8.35
Danish39.7151.37+11.66
English43.6252.45+8.82
Math & Code34.9741.86+6.90

The matched HRM outperforms the baseline across all four groups. This provides evidence that the HRM/PrefixLM/target-only recipe is effective at this stage of training. However, it should be noted that this does not isolate a potential effect of the PrefixLM attention mask and training objective, both of which differ from a standard causal Transformer. We will report the baseline's full configuration and evaluation protocol in the final version and will not present this comparison as definitive proof that recurrence alone drives the gains. These results are matched by training compute. We are running further ablations on the exact architectural mechanisms that contribute to the gain. We can confidently report already now that the effects of the different architectural aspects compound.

Response to Questions

Q1. When matched on the same compute and inference budget, what is the advantage of using this recurrent depth architecture over a standard Transformer baseline? As reported above, the computational depth of the architecture allows for superior performance compared to a matched Transformer. Furthermore, as demonstrated in this work, Mimir competes with models several times its size in terms of parameters.

Q2. The current training mix includes several components (recurrent-depth model, SDG datasets, and continual training); ablations to isolate these effects would be helpful. We agree that further ablations would provide valuable insights. However, even though Mimir is a relatively small model, conducting a comprehensive series of such ablations would be computationally out of our budget. This would require at least $20K in compute for a basic ablation.

Q3. As noted in the limitations section, training from scratch with reasoning data does not replace the need for broad knowledge acquired via next-token prediction cross-entropy loss. It is unclear why one would choose to train from scratch on instruction/reasoning datasets directly.

The motivation to train directly on instruction and reasoning datasets was introduced in HRM-Text. As described in our work, we formulate multiple pre-training tasks as post-training tasks to enhance language modeling capabilities, making the model more robust in tasks such as Grammatical Error Correction (GEC). Training directly on post-training data rather than a massive general corpus is motivated by two factors: i) limited accessibility to openly licensed data at pre-training scales, and ii) compute affordability, as it is significantly cheaper to forego traditional large-scale pre-training. We chose not to initialize from the HRM-Text checkpoint because i) we have added a tokenizer with chat-template introducing capability of multi-turn conversations and ii) not all of the Sapient data is, in our view, permissible for use. Because i) we changed the tokenizer ii) not all of Sapients data is permissible in our opinion

Thank you for adding a transformer baseline but I am still not convinced about the use of recurrent-depth model for this study and like authors said PrefixLM attention mask and training objective both are still not ablated clearly, so my first and second question about isolation is not addressed. Also 3rd question about using reasoning data from scratch, I still think there is sufficient available pertaining data with good license that can be used Common Pile, nemotron-cc and others a lot of them are available with good license. I will maintain my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

With the discussion period coming to an end, we use the opportunity to thank all reviewers once again for their insightful feedback.

We are particularly encouraged by reviewers noting the value of the “curated training corpus and SDG pipelines under data permissibility constraints” (9DXz), that we “train a small language model from scratch and achieve competitive performance” (9VAT), and that “the models match or surpass much larger frontier baselines, demonstrating that competitive performance is achievable without web-scale or non-permissible data.” (Zwpo).

A concern mentioned by two reviewers (9DXz, Zwpo) was whether the gains are only driven by our advances in synthetic data pipelines or also by architectural choices (recurrent depth, chat template, prefixLM, dynamic learning rate dividers).

To alleviate this concern, we have trained a data- and compute- matched standard transformer (as requested in the reviews) for one epoch on the entire dataset. Our results from this new experiment show that the recurrent depth architecture yields substantial advantages over a standard transformer (Unweighted average score across all 3 categories: 43.2 Mimir HRM-Text vs 31.9 compute/data-matched Standard Transformer). This replicates the findings of Wang et al. (HRM-Text; 2026) also for our modified architecture, data mix, and training setup.

We have updated the paper to include these new results on the matched Transformer baseline (Section 5, Table 5, Figure 2) and have revised abstract and contributions accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.