acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Parameter Monolith: Modular Language Modeling with Frozen Memories, Executable Skills, and Residual Assembly

Abstract

Language-model systems can separate contextual computation, persistent storage, and exact execution instead of updating all capabilities through one shared parameter system. We investigate FEM-ASM, a finite-element-method-inspired organization in which independently constructed document states and deterministic executable skills contribute typed proposals to a shared language-model state. An explicit residual operator reconciles proposals attached to common interface nodes. We evaluate this organization through controlled experiments and negative results rather than claiming a physical finite-element formulation of language. An attention-free Multi-Mesh prototype learns causal language modeling but does not establish competitive general capability. A versioned store contains 52,809 reconstructive memory elements near a 1.7-billion-floating-value budget; reconstruction is incomplete, with approximately 75% token accuracy. Support-aware lexical indices make these elements addressable under provenance-controlled query construction. For executable arithmetic, positional result observations substantially improve neural rendering relative to a repeated global result vector, and output substitutions change the model's preferred answer. A bounded attachment demonstration further measures the effect of making selected evidence available, without establishing the utility of loading an entire multi-billion-value store. The results support a separation of storage, execution, and neural coordination, while identifying unresolved limitations in question-only retrieval, unrestricted answer generation, and end-to-end efficiency.

Primary area:
generative models
Keywords:
LLM, fixed coordinates, immutable memories, Skills VM, ASM, FEM, retrieval, growth

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pnFPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a modular language model architecture called FEM-ASM. It decouples contextual computation, external knowledge storage, and deterministic execution, and integrates information from different modules into neural states through typed residual assembly. The paper also investigates the feasibility of combining different components and discusses the limitations of the current implementation.

Critical strengths
  1. The paper explores a new architecture that coordinates different types of external modules through a unified interface, providing a new perspective on how language models can be organized.

  2. The experiments involve multiple components, and the paper clearly discusses the limitations of the current implementation.

Critical weaknesses
  1. The core novelty and motivation are insufficiently justified. The paper does not provide enough evidence to demonstrate the unique advantages of the proposed architecture over existing approaches.

  2. The scope of this research is somewhat too broad, while the experimental validation is insufficient. The paper also lacks a clear storyline, making it difficult for readers to follow the main idea and understand how the different experiments support the central claim.

Questions for authors
  1. The motivation of this paper is not sufficiently clear. Why is such a modular integration architecture necessary? Current LLMs can already use various external tools and knowledge bases. In this context, introducing a new architecture seems somewhat unnecessary without demonstrating its unique advantages over existing approaches.

  2. The paper may need a clearer storyline. The current presentation feels more like an exploration of several relatively independent research directions rather than a coherent investigation of a central problem. Organizing the paper around a specific research question might make it easier for readers to follow. This would also help clarify the core contribution and how each experiment supports the authors' main argument.

  3. The paper uses too much defensive writing. Although this helps clarify the scope of the claims, repeatedly emphasizing what the paper does not claim tends to interrupt the flow of reading and makes the paper harder to follow.

  4. The current experiments mainly demonstrate the feasibility of individual components under controlled settings. More comparisons with existing architectures may be needed to demonstrate the advantages of the proposed framework. For example, the external memory mechanism lacks a direct comparison with standard raw-text retrieval approaches, making it difficult to evaluate whether the additional architectural complexity is justified.

3 comments
AuthorsOpenReview ↗

Response: coherent system contribution and the role of controlled experiments

Thank you for recognizing the unified interface for heterogeneous modules and for pointing out that the original narrative obscured it. Your observation about defensive writing was particularly useful: we have replaced repeated negative qualifications with a direct architectural explanation and a consolidated limitations section.

The motivation is an integration problem, not a claim that existing LLMs cannot use tools or knowledge bases. A document state, an execution result, and a contextual neural field have different update and verification requirements. FEM-ASM makes their boundaries explicit: local construction, versioned publication, support-aware readout, and shared neural assembly.

The architectural contribution is their combination in a working system. Memory workers do not update the core or synchronize document-state gradients. Published versions are consumed through fixed snapshots. Memory and VM observations use type-level rather than document-specific adapters. The revised algorithms describe these operations and the return path from external execution to neural output.

The experiments now follow this structure. Reconstruction evaluates retained source information; retrieval evaluates addressability; positional-readout comparisons evaluate access and rendering; fixed-core attachment evaluates the effect of newly available evidence. These are complementary tests of interfaces along one path, rather than independent proposals for four different systems.

We have also added the same-corpus 100B-token Transformer reference and retained explicit resource accounting. A matched raw-text comparison and an ablation against simpler injection operators remain open; we do not claim that the new reference resolves them.

We would appreciate reconsideration of the contribution as an implemented modular architecture with controlled interface evidence, rather than as a proposal for a superior standalone language model. The revision makes that intended contribution substantially more explicit.

Thank you for your explanation. However, it does not change my evaluation, and I will retain my score.

AuthorsOpenReview ↗

Clarification of the contribution being assessed

Thank you for considering the revision. We understand that you retain your recommendation.

For the final assessment, we would like to distinguish two claims. The paper does not establish that residual assembly is the best-performing integration operator. It does implement and evaluate an architecture in which document-state construction, artifact publication, and neural consumption have separate lifecycles, while memory observations and execution results enter the same recurrent computation.

The experiments examine successive interfaces in that path: retaining source information, locating an element, exposing its content, and rendering its contribution. Their component-level measurements make the integrated system diagnosable; they are not intended as unrelated proposals.

If the remaining concern is insufficient novelty or significance of this organization, we understand that judgment as distinct from a technical error in the reported construction or measurements. We would appreciate that distinction being reflected in the final assessment.

Reviewer_27Z9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a way to connect a language model with separately built document storage and tools. The proposed system, FEM-ASM, lets these parts provide information to a shared model without having to train or update everything together. The authors test how well the system stores and retrieves documents, uses calculation results, and produces answers.

Critical strengths
  1. In the proposed system, updating a document or replacing a tool should not require retraining the language model.
  2. The paper separates finding information, recovering it, doing the calculation, and writing the answer. This helps show where things go wrong.
Critical weaknesses
  1. I found reading this paper really difficult. The abstract reads like a set of experiments rather than one clear scientific finding. I is not clear from abstract what FEM-ASM actually does, which experiment tests its central idea, and why the results matter?
  2. Language models already use document stores and external tools. The introduction does not clearly explain what is different here or what existing systems struggle with.
  3. Why store documents this way instead of keeping the text? This needs an explanation, especially since the abstract says reconstruction gets only about 75% of tokens right.
  4. The introduction is hard to follow. Terms like “typed residual assembly” and “consistency guarantee” need simple explanations or examples. The Unicode paragraph feels disconnected from the main goal, and the introduction should briefly say what worked, what failed, and what was learned.
  5. The paper reports (section 5.2) 1,536 bytes per memory cell compared with 128 bytes for the original token IDs, before adding indices and caches. Without comparisons to simpler approaches, such as storing the original text and passing tool results directly to the model, it is hard to judge whether the extra storage and complexity are worthwhile.
  6. Only 1 of 2,910 (Table 2) tested blocks was reconstructed completely correctly. This is more concerning than the headline 75% token accuracy, especially for recovering exact facts or quotations.
Questions for authors

See the weaknesses!

3 comments
AuthorsOpenReview ↗

Response: architectural purpose, raw-text comparison, and reconstruction fidelity

Thank you for recognizing the separation between finding information, recovering it, executing a computation, and writing an answer. We have made that separation part of a clearer architectural account.

What differs from ordinary external-resource use?
The contribution is not external memory or tool use alone. It is the implemented combination of independently optimized document states, immutable publication and snapshots, and a common typed internal interface for memories and execution results. Components retain different construction procedures while participating in the same neural update mechanism.

Why reconstructive states rather than raw text?
The cells investigate a particular internal interface: a frozen decoder exposes position-specific states that the core consumes through shared adapters. The current representation is larger than uint16 source tokens, and we retain that accounting. We do not present storage compression as its advantage. Its measured value in this study is as an interface to independently constructed document state.

A matched raw-text reader remains an important missing comparison. The added 100B-token Transformer reference addresses standalone language-model quality, not this memory-interface question; the revision keeps those comparisons separate.

What does incomplete reconstruction establish?
We agree that 74.97% token accuracy cannot be read as reliable exact-block recovery. The full-block result remains visible. The readout comparison provides additional evidence: codec-position observations achieve NLL 4.4729 and 48.60% token accuracy, versus 6.6331 and 12.15% for the repeated cell. These are readout diagnostics, not proof of exact fact preservation.

What changed in presentation?
We rewrote the abstract and introduction, added two explanatory figures, moved the Unicode details out of the central narrative, and organized the results around the path from construction to neural use. Limitations are consolidated rather than repeatedly interrupting that explanation.

AuthorsOpenReview ↗

Why store a prepared neural representation rather than only token IDs?

We would like to make the intended role of the memory representation more concrete.

The stored cell is a prepared neural representation for the consuming solver, not an alternative compressed text file. Its purpose is to move document-specific computation into an independent construction stage and expose the result through a reusable internal interface.

The distinction is analogous to a materialized database view: the prepared representation may occupy more space than its source while serving a different computational role. Accordingly, the comparison between 1,536 bytes of cell state and 128 bytes of token IDs measures archival overhead, but does not by itself determine the cost or usefulness of consuming that information.

A complete comparison must include both paths:

  • Text path: source storage, retrieval, encoding or prefill, retained state where applicable, and answer generation.
  • Prepared-state path: local construction, cell and index storage, decoding or cached readout, adaptation, and answer generation.

Both paths can reuse computation. Their relative utility depends on reuse frequency, update cost, fidelity, and serving behavior. The current paper establishes the prepared-state interface and measures its use; it does not infer an end-to-end efficiency gain from cell size.

The reconstruction metric has a similarly specific role. Exact recovery of every token is necessary for a lossless archive, but is not a prerequisite for every useful intermediate representation. Conversely, token reconstruction alone does not establish preservation of task-relevant facts. We therefore evaluate consumption separately: positional codec observations improve readout relative to a repeated cell, and selected evidence improves predictions with unchanged core weights.

The architectural objective is to make independently constructed document state available for contextual computation through the same integration mechanism used for execution results. Reconstruction, retrieval, and rendering measurements identify different properties of that complete path.

I thank the authors for their rebuttal. After considering their response, I continue to share the other reviewers’ concerns about the paper’s motivation and readability. The paper needs substantial revision to clarify why the problem matters and make the argument easier to follow before it is ready for publication at this conference. I therefore maintain my original evaluation.

Reviewer_DGyMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors study the modular organization of language models: that Reconstructive Memories, Executable Skills, and Residual Assembly are three parts that are separately measurable. However, the paper as a whole is hard to follow and is clearly below the standard of a normal research paper. The research questions are not well motivated, the claimed contributions are confusing, and the technical sections are badly presented. Overall, I recommend rejection.

Critical strengths

S1. The authors study the modular organization of language models: that Reconstructive Memories, Executable Skills, and Residual Assembly are three parts that are separately measurable

Critical weaknesses

W1. The research question is not well motivated in the introduction.

W2. The claimed contributions are confusing. The current version shows that the authors propose components A, B, and C, yet we cannot find any description of their detailed functionality.

W3. The technical sections are badly presented. For example, in Section 3.1, several symbols are introduced without ever being defined.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response: motivation, component functionality, and notation

Thank you for identifying the lack of a clear architectural explanation and the difficulty following the technical presentation. We have revised the introduction and architecture sections around one question:

How can independently constructed memories and executable results participate in shared neural computation without sharing their construction or optimization process?

The components now have explicit responsibilities:

  • Reconstructive memory: locally optimized document state produced under a frozen codec and published as a versioned artifact.
  • Executable skill: a bounded program whose result is computed independently of neural weights.
  • Residual assembly: a shared interface that maps their observations to proposals at specified neural nodes and reconciles those proposals before learned contextual processing.

The architecture figure shows the complete production-to-output path. The appendix provides algorithms for its principal operations, including the return from execution to neural consumption.

We also audited the notation. Each observation is now defined by its type, source state, sample index, target node, availability boundary, weight, and provenance. The adapter, assembly rule, consistency substep, and learned correction are described separately. The implementation’s numerical acceptance rule and the scope of the consistency guarantee are documented explicitly.

These changes directly address W1–W3. We would appreciate your reassessment of whether the revised explanation makes the component functionality and central research question sufficiently clear. If a specific soundness concern remains beyond the presentation issues, identifying it would allow us to respond precisely.

The key issue with your submission is captured well in the sentence you emphasize:

How can independently constructed memories and executable results participate in shared neural computation without sharing their construction or optimization process?​

This sentence piles term upon term without making clear what any of them means, and after reading it I still have no idea what is actually being claimed.

The beginning of Section 3 is still really hard to follow.

  • sample b: a sample of what?
  • node i: a computation node, a graph node, or a sequence position?
  • iteration k: no iterative process is introduced before this point.
AuthorsOpenReview ↗

Concrete computation and definitions of the state indices

Thank you for identifying the exact point where the explanation remains unclear. The intended operation is the following:

We prepare document representations before a query is processed, then let a language model consume those representations directly in its hidden states. A computed tool result enters through the same mechanism.

For example, a worker takes a 64-token document block, keeps the shared codec weights fixed, and optimizes a vector representing that block. Later, the decoder exposes observations from the stored vector. A shared adapter maps each observation to the language model’s hidden-state width. The model combines these observations with its current states and performs repeated updates before predicting tokens.

The indices mean:

  • bb: one token sequence in the current minibatch;
  • ii: a token position within that sequence, used as an attachment location;
  • kk: the number of completed internal solver updates within one forward pass, before output prediction. It is not a generated-token index.

Thus, Zb,ikZ_{b,i}^{k} is simply the hidden vector at token position ii of batch sequence bb after kk updates. The same local and correction modules are reused at each update.

If an observation proposes vector zez_e at that position, its residual is ze−Zb,ikz_e-Z_{b,i}^{k}. Multiple observations contribute a weighted, normalized sum. The model first moves toward their weighted mean and then applies a learned contextual correction.

The architectural claim is that document representations and execution results can be constructed separately and used through this shared internal interface. The fixed-core attachment experiment tests one instance of this: changing the available evidence changes predictions without updating the consuming model’s weights.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision summary: architectural contribution, implementation, and comparative evaluation

We thank the reviewers for identifying a central presentation problem: the original manuscript made the individual experiments easier to see than the architecture connecting them. We have substantially revised the paper to make that architecture explicit.

The contribution is a concrete organization of independently produced external state and shared neural computation. Document workers optimize local memories under a frozen codec and publish immutable versions. Consumers use fixed snapshots. Executable skills return deterministic results. Both enter the neural core through type-shared adapters, explicit supports, and the same residual-assembly interface. Document-specific construction does not require document-specific adapter parameters or updates to the core.

The revision adds:

  • An architectural overview and a second figure explaining global versus positional readout.
  • Explicit observation fields, adapter equations, solver updates, and compatibility boundaries.
  • Algorithms for assembly, recurrent reconciliation, memory construction, publication, snapshots, execution, and the neural–external–neural information cycle.
  • A same-corpus comparison with AB-EXT Learned, a 1.812B-parameter Transformer trained with a 100B-token budget. Its training budget is not matched to Multi-Mesh, and we label it as a reference rather than a controlled architecture-only comparison.
  • Expanded implementation and evaluation documentation, including resource accounting and failure boundaries.

The main empirical findings are unchanged: positional observations improve the tested return interfaces, and selected evidence improves predictions with unchanged core weights. The revised narrative explains how these findings inform the architecture rather than presenting them as unrelated component demonstrations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.