acceptodds
Under review as a conference paper at ICLR 2027

Nesting Is All You Need: Recursive Memory Plane with Latent Specialization

Abstract

Most language models treat depth as disposable computation: intermediate representations are transformed and discarded until only a terminal state remains. We introduce **Recursive Memory Plane**, a language-model architecture built around nested specialization: Primary (P), Intermediate (I), and Association (A) cells are structurally specialized for local, contextual, and global computation, forming a persistent P→I→A hierarchy whose entire representational trajectory remains jointly addressable. Rather than recur only over the terminal hidden state, each recursive step performs a **reason–readdress–integrate cycle**: a reasoning state (R) queries the full memory plane, then a discourse state (D), conditioned on the updated reasoning state, independently re-queries the same plane to form the next prediction state. **LatentBank** further adds compact, persistent I- and A-specialist states, preserving nested specialization alongside recursion. This design yields an unexpected insight: **recursion is most valuable during training, while shallow inference performs best**. At 220M scale, LatentBank adds only 0.23% parameters; training with three recursive steps and inferring with one achieves a **34.66% raw mean across nine benchmarks**, versus **33.87% for the strongest parameter matched baseline**. Eliminating inference recursion still reaches 34.52%, whereas training without recursion falls to 32.99%. These results suggest a different design principle: **nested specialized computation, preserve depth as memory, recursively re-address the full representational trajectory, and use deep training to produce powerful shallow inference.**

Primary area:
foundation or frontier models, including LLMs
Keywords:
recursive memory, nested learning, language modeling, latent representations, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uVZnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a language-model architecture featuring 31 interleaved cells categorized as Primary, Intermediate, Hub, and Association cells. Local convolutions, windowed attention, and global attention operate on persistent, specialized states; incremental updates are propagated via "Telscoping Residual Inheritance," while the output of each cell is retained as a form of "deep memory." Subsequently, a terminal Association operator performs two sequential "deep reads" to update the model's reasoning and discourse states. A "LatentBank" component introduces compact, specialized states used to modulate the attention inputs within the backbone. A key experimental finding is that employing three-step recursion during training enhances downstream performance, even when inference is restricted to zero or one recursion step. At a scale of 220 million parameters, a configuration using three-step training and one-step inference achieves an average score of 34.66% across nine tasks. The paper also extends these "three-seed comparisons" to model scales of 400 million and 800 million parameters.

Critical strengths

This idea is clear and easy to follow; the three components of the structure: Y, Z, and LatentBank, are also clear.

This finding that increasing the number of recursive steps during training allows for the retention of most performance gains, even when the number of steps is reduced during inference, is valuable. And the experimental results demonstrate a degree of stability; improvements were observed across three different model scales and multiple random seeds, indicating that this advantage is not merely an artifact of a single experiment.

Critical weaknesses

It remains unclear whether full-depth memory is necessary. Reading only the final state yields an average accuracy of 34.74%, while using the complete trajectory results in 34.66% (Table 1). By training a model that has access only to the final state, one can verify whether full-depth memory is essential during the training process.

Some performance gains may stem from the additional computational effort during training. With the number of recursion steps fixed at zero during inference, increasing the number of recursion steps during training from zero to three improves the average accuracy from 32.99% to 34.52% (Table 3(c)). However, having the same number of parameters and the same volume of training data does not necessarily imply an equivalent computational cost. Conducting comparisons under an "equal-compute" setting helps to disentangle the advantages inherent to the architecture itself from the gains derived from additional computation.

Questions for authors
  1. Why is the accuracy in MATH of Table 1 all 0? It should be clearly explained the reason for this in the main text.

  2. The author states in the paper that "Our design is also motivated by evidence that biological language processing is distributed across differentiated systems," yet this point is not addressed in the subsequent method section—which is quite confusing.

  3. "Shallow inference after recursive training"—can this paragraph can further analysis to understand why performance on certain tasks falls short of the baseline?

  4. There are a few typos: at the end of the "Inference profiling" paragraph, type an extra ".".This description in Table 3: "Panel(a)".

4 comments
AuthorsOpenReview ↗

Rebuttal 1 - Thanks for your valuable feedback

We appreciate reviewer for spelling out clarity of our architectural contributions, value of recursive training for shallow inference, and consistent results across three model scales and independent seeds. We especially appreciate positive soundness assessment.

Due to character limit, we provide our responses in two parts.

We’ve conducted additional experiments addressing two principal concerns. trained terminal only controls at 220M, 400M, and 800M, and training resource accounting. We also clarify MATH results, neuroscience motivation, and task specific limitations.

1. Is full depth memory necessary during training?

Our new experiments provide direct evidence that full depth access improves training outcomes.

You correctly distinguish existing terminal only inference intervention (Appendix E, Table E.1) from a model trained without full depth retrieval. former achieves 34.74% versus 34.66% because it uses a checkpoint originally optimised with access to all 31 depth states. It cannot determine whether this access contributed during training.

To address this question, we trained terminal only recursive models at all three capacities. These retain the 31 cell specialist backbone, TRI, LatentBank, and recursive R/D updates, but restrict both recursive reads to Y31 throughout training across 3 seeds per param config.

New results (nine-task mean accuracy)

ScaleFull RMPTrained terminal-onlyGain
220M34.6633.68 ± 0.12+0.98
400M37.8136.42 ± 0.60+1.39
800M39.8538.24 ± 0.52+1.61

Full trajectory training consistently achieves higher downstream accuracy at every tested capacity. This is precisely comparison needed to distinguish training time utility from inference time redundancy.

The surprising terminal only inference result is now making kinda sense. full depth access during optimisation improves learned representations, yet those representations can subsequently support effective terminal only inference.

Our held out language modeling experiment provides complementary evidence. At Z=3, full trajectory retrieval achieves NLL 3.7951 (PPL 44.48), whereas terminal only retrieval achieves NLL 3.8266 (PPL 45.91). Thus, complete depth access also retains measurable utility for next token likelihood.

2. Training compute and architectural attribution

We agree that parameter matching and equal processed tokens don’t imply identical training computation. We additionally inspected our W&B training logs.

220M resource measurements

ConfigurationTrain ZMean GPU memoryMonitoring span
TRM111.63 GiB6.43 h
mHC013.15 GiB8.36 h
RMP012.65 GiB7.07 h
RMP113.32 GiB8.15 h
RMP313.89 GiB10.42 h

These measurements explicitly expose the increased resource requirements of recursive training.

Importantly, trained terminal only controls provide a kind of targeted architectural comparison than external parameter matching i.e. they retain same backbone and recursive structure while changing information supplied to recursion.

Our inference analysis separately demonstrates that, at 220M, RMP with Z=0 achieves 34.52% at 16.22k tokens/s, compared with mHC's 33.87% at 15.93k tokens/s under same full sequence profiling protocol. Hence deeper training can yield a useful shallow inference operating point, while additional training expense remains acknowledged.

<< CONTINUES IN FOLLOWING PART >>

3. Why is MATH accuracy zero?

We appreciate spotting this one for clarification.

Our evaluation uses MATH 500, zero shot greedy generation, a maximum of 128 generated tokens, and normalised exact matching of generated boxed answer against the reference boxed answer (Appendix B.4).

Under this protocol, all compared models at 220M, 400M, and 800M obtain 0.00%. This is measured outcome of specified exact answer evaluation.

Our models are relatively small pretrained language models evaluated without task specific mathematical fine tuning.

We agree that main text should explicitly explain evaluation setup and avoid interpreting zero exact match accuracy more broadly than warranted and we will make these clarifications accordingly.

4. Neuroscience motivation

We agree that distinction between architectural inspiration and biological correspondence should appear more clearly in Section 2.

The neuroscience evidence motivates an organisational prior of interacting, differentiated language processing systems, not a claim that RMP reproduces biological language networks.

This prior is instantiated computationally through:

  • Primary cells: local causal convolution for short range information.
  • Intermediate/Hub cells: windowed attention for contextual processing and interaction with global representations.
  • Association cells: global causal attention for long range integration.
  • TRI and LatentBank: persistent information exchange while retaining differentiated specialist states.
  • Recursive Memory Plane: explicit access to representations at different stages of this hierarchy.

These roles are our engineering choices. We will connect this motivation to method more explicitly.

5. Why do some tasks underperform baselines?

We agree that overall aggregate should not obscure heterogeneous task level behavior.

At 220M, RMP performs well on BLiMP (76.59%) and BoolQ (59.57%), but trails TRM on ARC Challenge (21.74% versus 23.41%) and GSM8K (0.99% versus 1.44%), and slightly trails mHC on OpenBookQA (15.80% versus 16.20%).

These results suggest that improvements in linguistic and contextual representations do not automatically translate into improvements in every form of multi step reasoning or knowledge retrieval.

The present experiments do not isolate a unique causal explanation for these deficits, and we will avoid making one. We will expand discussion of task dependent behavior and make clear that Z=1 is best observed depth for reported nine task aggregate, not a universally optimal inference setting for sure.

Indeed, WikiText-103 perplexity continues improving through Z=3 (58.08 at Z=0, 50.83 at Z=1, and 44.48 at Z=3), establishing an objective dependent trade off.

As an additional robustness check, strict BBH final answer rescoring preserves RMP's lead at all three capacities, and nine task aggregate remains highest for RMP after replacing original BBH results.

6. Presentation corrections

We thank reviewer for identifying typographical issues. We will remove the duplicated period in the inference profiling paragraph and correct Table 3 panel description. We will also strengthen main text explanation of MATH evaluation, clarify neuroscience to method connection, and foreground limitations of downstream aggregate.

To conclude our responses, We hope new results directly address reviewer's central architectural question i.e. full depth memory is not required to preserve shallow downstream accuracy after full memory training, but trained terminal only models consistently underperform full RMP at 220M, 400M, and 800M.

We sincerely appreciate constructive suggestions and hope we have addressed reviewer’s concerns accordingly. Thank you very much for your time reading and reviewing our paper and sharing your feedback with us.

Thank you for your response which resolves my questions. I keep my original score.

Glad to read that we’ve addressed your concerns. Thank you 🙏🏼

Reviewer_LNbHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines nested local, windowed, and global operators with retained depth representations, recursive R/D updates, and compact LatentBank states. Its central empirical finding is that recursive training can improve downstream performance even when inference recursion is reduced or removed. Experiments compare the architecture with parameter-matched baselines across three model scales.

Critical strengths
  1. The three-seed comparisons at 220M, 400M, and 800M provide useful evidence that the reported aggregate ordering is reproducible across the tested settings.

  2. The paper appropriately distinguishes independently trained controls from interventions on an existing checkpoint. The training-depth controls reveal an interesting separation between training and inference computation.

  3. The appendix transparently includes terminal-read interventions and held-out language-modeling results, allowing readers to assess limitations of the main interpretation.

Critical weaknesses
  1. Appendix B.4 scores BBH by whether the normalized target occurs anywhere in the generated completion. This can accept an incorrect final answer containing the target or a response listing multiple candidates. Applying the same rule to every model does not guarantee equal bias. The impact on the reported comparisons needs to be established through rescoring.

  2. The training contribution of full-trajectory memory is not isolated. Table E.1 shows that terminal-only reads preserve downstream performance at inference, but this does not determine whether full-depth access was necessary during training. A trained terminal-only control and clearer training-compute accounting would address the central attribution question.

  3. The shallow-inference advantage is metric dependent. Table E.5 reports perplexities of 50.83 at Z=1 and 58.08 at Z=0, versus 44.74 for mHC. Although the appendix acknowledges this trade-off, the main conclusions should consistently qualify shallow inference as preferable for the reported downstream aggregate rather than language-modeling quality generally.

Questions for authors
  1. Can you rescore the existing BBH generations using task-appropriate final-answer extraction, report the 27 subtask results, and recompute the aggregate comparisons? Please clarify normalization and audit responses containing competing answers.

  2. Do you have a terminal-only recursive-read control trained from scratch under a comparable compute budget at a representative scale?

  3. How should the shallow-inference claim be interpreted alongside Table E.5? Was inference depth selected using validation evidence independently of the reported benchmark outcomes?

1 comment
AuthorsOpenReview ↗

Appreciate your feedback on our paper

We thank reviewer for recognising three scale results and training/inference distinction. We conducted terminal only training from scratch at 220M/400M/800M and strict BBH rescoring of all four model families at every scale directly addressing two major concerns.

1. Full depth memory during training

Table E.1 tests terminal only inference on a full memory trained checkpoint. Our new from scratch controls retain 31 cell backbone, TRI, LatentBank, and R/D recursion, but feed only Y31 to both recursive reads throughout training.

ScaleFull RMPTrained terminal-onlyFull-depth gain
220M34.6633.68 ± 0.12+0.98 pp
400M37.8136.42 ± 0.60+1.39 pp
800M39.8538.24 ± 0.52+1.61 pp

Full depth training scores higher at all scales, while terminal only inference after full memory training preserves downstream accuracy (34.74 vs 34.66 at Z=1). This supports a role for full depth access during representation formation. At Z=3 full retrieval also improves NLL versus terminal only inference (3.7951 vs 3.8266; Table E.5).

2. BBH scoring

Our substring rule could accept competing answers. We rescored principal BBH comparisons using strict final answer matching:

ScaleTransformerTRMmHCRMP
220M53.22±0.7657.34±0.8257.75±0.5358.72±0.64
400M63.72±0.7370.35±0.9669.24±0.8873.06±0.93
800M72.09±0.4478.96±0.7478.76±0.5680.42±0.49

Strict scoring lowers original scores, but RMP remains highest on BBH at all three scales. Substituting these strict results into nine task aggregate, with the other eight tasks unchanged gives:

ScaleTransformerTRMmHCRMP
220M31.8332.8433.7834.55
400M34.2635.6036.2637.68
800M36.4237.9838.8539.83

RMP still leads mHC by +0.76/+1.42/+0.98 points. Even excluding BBH entirely, RMP leads at all scales (+0.73/+1.11/+0.90).

3. Meaning and selection of shallow inference.

We agree with proposed qualification. Z=1 is best observed operating point for reported nine task downstream aggregate. On WikiText-103, full RMP improves from PPL 58.08 (Z=0) to 50.83 (Z=1) to 44.48 (Z=3); Z=3 also has lowest measured NLL among compared 220M settings (3.7951 vs 3.8009 for mHC). shallow RMP PPL is worse than mHC and we will state this explicitly.

4. Training compute transparency.

We agree that parameter/token matching is not FLOP matching. Our W&B logs give following 220M mean GPU allocated memory / observed monitoring span: TRM 11.63 GiB / 6.43 h, mHC 13.15 / 8.36 h, RMP Ztrain=0 12.65 / 7.07 h, Ztrain=1 13.32 / 8.15 h, and Ztrain=3 13.89 / 10.42 h. We acknowledge additional training expense and make no compute optimality claim.

In summary, trained terminal only controls now favor full depth memory at all three scales, while strict BBH rescoring preserves RMP's lead. We clarified metric dependent shallow inference claim and reported measured resource use without claiming compute parity. We hope we have addressed reviewer’s concerns.

As a finishing note, we kindly invite the reviewer to compare the Liquid AI’s LFM3 tri-hybrid architecture (released after our submission) with our proposed model, which strikingly share the same 3 components as ours (we were surprised by the level of overlapping in broader concepts).

Thank you again for your valuable feedback which helped improve our research.

Reviewer_s1boOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Recursive Memory Plane (RMP), which combines nested specialist modules, retrieval over intermediate representations, and compact latent states. It reports consistent gains across three model scales and finds that recursive training can improve downstream performance with shallow inference.

Critical strengths
  1. The separation between training and inference recursion is interesting. Training with three recursive steps improves zero-recursion downstream performance from 32.99 to 34.52 (Table 3).
  2. The main results are consistent across three scales and three training seeds. The appendix also provides useful retrieval interventions and language-modeling evaluations.
Critical weaknesses
  1. The overall motivation for full-depth memory is unclear. Terminal-only retrieval matches full-trajectory retrieval on the downstream aggregate, scoring 34.74 versus 34.66 (Table E.1). A terminal-only model trained from scratch is needed to show whether the proposed memory plane adds value beyond simpler recursive computation.

  2. The shallow-inference conclusion depends on the metric. RMP perplexity improves from 58.08 to 44.48 as inference recursion increases from zero to three steps (Table E.5). Shallow inference preserves downstream accuracy, but its language-modeling quality is worse than the Transformer and mHC baselines. The main claims should reflect this limitation.

  3. The BBH scoring rule may overestimate accuracy. The evaluator accepts an output whenever it contains the target string, which could count outputs containing multiple candidate answers as correct (Appendix B.4). Strict final-answer matching is needed to verify the reported ranking and preferred inference depth.

  4. Training computation is not controlled. Matching parameters and training tokens does not match the cost of recursive training. A comparison under the same training-compute budget would clarify whether the gains justify the additional computation.

Questions for authors
  1. Does full-depth retrieval outperform terminal-only retrieval when both models are trained from scratch?
  2. Do the main conclusions hold under strict BBH scoring and matched training-compute budgets?
2 comments
AuthorsOpenReview ↗

Rebuttal part 1 - Your feedback helped improve our paper

We thank the reviewer for constructive assessment and for recognising the significance of training/inference recursion separation, three scale consistency, and additional analyses. Due to character limit, we will post our responses in two parts.

We took soundness concerns seriously and conducted two targeted follow up studies in the last week: (1) independently trained terminal only recursive controls at all three model scales, and (2) strict BBH rescoring for every principal model family at all three scales. These experiments directly address first and third critical weaknesses.

1. Does full depth retrieval outperform terminal only recursion when trained from scratch?

Our new training controls provide consistent evidence that it does.

We agree that same checkpoint terminal state intervention in Table E.1 can’t establish contribution of full depth access during optimization. That intervention establishes an inference time result i.e. after full memory training, learned representations support accurate terminal only inference.

To address reviewer's question, we additionally trained terminal only recursive models from scratch at 220M, 400M, and 800M. These controls retain nested specialist backbone, TRI, LatentBank, and R/D recursion, but restrict both recursive memory inputs to terminal representation Y31 throughout training.

our new results are:

ScaleFull RMPTerminal-only trainingFull RMP advantage
220M34.6633.68 ± 0.12+0.98 pp
400M37.8136.42 ± 0.60+1.39 pp
800M39.8538.24 ± 0.52+1.61 pp

The paradigm is directly tested as we claim here. full depth access during training produces higher observed downstream performance than trained terminal only recursion at every tested capacity.

We invite kindly the reviewer to also note this finding is compatible with Table E.1, where terminal only retrieval at inference reaches 34.74% compared with 34.66% for complete reader. Hence, two experiments address distinct questions:

  • Training time access: trained terminal only models underperform full RMP training across all three scales across 3 seeds.
  • Inference time access: Once trained with full trajectory, model can preserve downstream aggregate accuracy using only Y31 which is also interesting in our view.

Our new findings support proposed recursive representation shaping mechanism i.e. access to heterogeneous depth representations during optimisation can improve representations subsequently available for shallow or restricted inference.

language modeling results provide complementary evidence. At Z=3, full trajectory retrieval obtains NLL 3.7951 and PPL 44.48, whereas restricting same trained checkpoint to Y31 gives NLL 3.8266 and PPL 45.91 (Table E.5). Hence, full depth retrieval also provides a measurable next token likelihood benefit at training recursion depth.

We believe these additional trained controls substantially address reviewer's main architectural concern.

2. Does the principal ranking survive strict BBH scoring?

Yes. We rescored BBH using strict final answer matching, and RMP retains highest mean BBH accuracy at all three scales.

We agree that original normalised substring matching rule could accept ambiguous responses. Although rule was applied consistently across architectures and explicitly documented in Appendix B.4, strict final answer matching is highly appreciated as reviewer has advised us.

new BBH results are:

ScaleTransformerTRMmHCRMP
220M53.22 ± 0.7657.34 ± 0.8257.75 ± 0.5358.72 ± 0.64
400M63.72 ± 0.7370.35 ± 0.9669.24 ± 0.8873.06 ± 0.93
800M72.09 ± 0.4478.96 ± 0.7478.76 ± 0.5680.42 ± 0.49

Strict scoring reduces reported BBH accuracies, as reviewer anticipated. Importantly, ordering remains favorable to RMP at all three scales. Its margins over the strongest BBH baseline are +0.97, +2.71, and +1.46 percentage points at 220M, 400M, and 800M, respectively.

We also recomputed principal nine task aggregates by replacing original BBH scores with new strict scores while retaining other eight benchmark results:

ScaleTransformerTRMmHCRMP
220M31.8332.8433.7834.55
400M34.2635.6036.2637.68
800M36.4237.9838.8539.83

The principal aggregate advantage survives strict BBH scoring at every tested model scale, with RMP margins over mHC of approximately +0.76, +1.42, and +0.98 percentage points.

This is also consistent with a separate robustness calculation excluding BBH entirely i.e. RMP still leads strongest external aggregate baseline across all three scales.

<< CONTINUES IN NEXT PART >>

3. Shallow inference and language modeling perplexity

We agree that preferred inference depth depends on evaluation objective and appreciate reviewer highlighting this distinction.

Our claim of strong shallow inference refers specifically to downstream benchmark performance after deeper recursive training. We don’t imply that Z=0 or Z=1 minimises next token cross entropy.

Existing WikiText-103 results in Table E.5 make this more even clearer:

RMP inference depthNLLPPL
Z=04.061958.08
Z=13.928550.83
Z=33.795144.48

downstream aggregate prefers Z=1, whereas language modeling objective improves through training depth Z=3.

At Z=3, RMP attains lowest measured NLL in this comparison, versus 3.8009 for mHC and 3.8328 for Transformer.

We acknowledge that shallow RMP inference has worse perplexity than those baselines. Our contribution is more precisely characterised as recursive training improving shallow downstream representations while allowing an objective dependent inference depth tradeoff.

4. Training compute matching

We agree that equal parameter counts and processed token budgets do not establish equal training FLOPs or GPU hours. To make observed resource requirements transparent, we additionally report GPU telemetry from our Weights & Biases (W&B) training logs, which will be released with experimental artifacts. The table below summarises corresponding 220M runs, including our training recursion controls.

220M training configurationTrain ZPer GPU allocated memory (GiB)W&B monitoring span (hours)
TRM111.636.43
mHC013.158.36
RMP + LatentBank012.657.07
RMP + LatentBank113.328.15
RMP + LatentBank313.8910.42

Memory values represent the mean allocated memory per GPU. Monitoring spans are calculated from first and last recorded timestamps of each run.

These measurements make additional resource cost of recursive training explicit. Within our RMP configurations, increasing training recursion from Z=0 to Z=3 is accompanied by higher allocated GPU memory and a longer observed monitoring interval. Full RMP Z=3 trace records average of 13.89 GiB on a GPU, compared with 13.15 GiB for mHC and 11.63 GiB for TRM.

5. Clarifying presentation and central claim

We appreciate presentation feedback and agree that original architecture introduces several interacting mechanisms.

The experiments now motivate a more straight interpretation:

  1. Recursive training is central: Training with Z=3 substantially improves shallow downstream performance relative to training without recursion.
  2. Full memory plane contributes during learning: Newly trained terminal only controls score below full RMP at 220M, 400M, and 800M.
  3. Full memory access can become redundant for shallow downstream inference: Same checkpoint terminal only retrieval preserves aggregate at Z=1, while full depth access remains beneficial for language modeling likelihood at Z=3.
  4. The main model ranking is robust to stricter BBH scoring: RMP retains the highest nine task aggregate at all three capacities.

We will foreground these summaries and more explicitly separate training time architectural effects, inference time retrieval dependence, downstream accuracy, and language modeling likelihood in presentation.

Closing remarks

We thank reviewer for identifying two important empirical gaps. We have now directly addressed requested trained terminal only comparison across all three tested capacities and strict BBH scoring concern across all principal model families and capacities.

new results establish consistent observed gains from full depth training over terminal only training (+0.98, +1.39, and +1.61 points), while strict BBH rescoring preserves principal aggregate RMP advantage (+0.76, +1.42, and +0.98 points over mHC).

We hope that we have addressed reviewer’s concerns accordingly and once again thank you a lot for your time reading our paper and sharing your feedback with us as it helps improving our paper a lot.

Comments on the paper

1 comment
AuthorsOpenReview ↗

🚨 Knowledge erosion in ML community

public class HelloWorld { public static void main(String[] args) { System.out.println("Hello World!"); } }

Dear PCs/SACs, ACs, esteemed reviewers and ML researchers,

We trust this message finds you well. We want to thank you for the efforts you have made during this ICLR cycle with the staggering number of submissions. I’ve been serving as reviewer and PC in these venues throughout last two years, and I have witnessed the good, the bad and alas more the ugly part in these venues.

We personally believe it is over for ML community and I am very sorry to communicate this message here. I will share this publicly and I hope this is a reminder that we need to do something otherwise we will be dragged into epistemically erosion of knowledge generation in this very critical domain and moment.

First of all, we want to challenge the concept of reviews being written from one source. We mean, these reviews are strikingly similar and I have seen in the community authors also raising the same concern over nitpicking which is pretty evident to anyone working in the field reflecting the AI generated nature of the text.

Please don’t get us wrong. We are not requesting to accept our paper. Please feel free to reject it. However, I hope what we share here will be worth of acceptance for generations. We are confident of what we have done, we are aware of its limitations, and we know what we have presented. In the sense, we were so overwhelmed by the release of Liquid AI's tri-hybrid architecture LFM3 after our submission which strikingly share the same kind of core mechanisms with our proposed RMP! i.e. the same kind of convolutions, sliding window attention (which is 256 tokens like ours) and global attention. We believe that RESPONSIBLE knowledge generation and contribution to the community matters much more than a badge of an acceptance in the venues and presenting a poster. Thanks to the arXiv and alphaXiv, these are also functional in the same level of the posters if not more than them.

What we want to communicate is the subtle line between AI generated (few shot if not one shot), AI+human collab, human only.

I will be honest, the last time I wrote the entire code myself from scratch was two years ago. With this great technology that we have now, I don’t think we need to spend time to write line by line myself, I can spend my valuable time in other areas. But certainly I review the code line by line. That also applies to the writing the paper. Using AI to write the paper and disseminate your results that you have spent 6 months in a small lab to achieve it is way different than writing a paper overnight!

What I want to communicate is this: there is a plague which I call it SYNTHETIC SUSPECT SYNDROME, leading to ungrounded rejection of the papers. I am sharing this experience from other venues including ICML, NeurIPS and AAAI where I have served as PC and reviewer. So how it works, the number of submissions skyrocket, reviewers believe most of them are AI generated (which is not wrong). And some Good Samaritan works also mix in between these AI generated/ irreproducible works being rejected on the basis of bias the reviewer already has. I mean, the paradigm has shifted and the reviewers are not looking in the papers the same way they used to in 4 years ago and before. They are already biased and suspicious of the paper. I have seen a lot of these cases on X other than these venues.

The sad part is that the reviewers give the paper to the AI (which I believe they shouldn’t as it is not published yet and authors have full right to maintain the confidentiality), and hit the prompt like “why this paper sucks?!”, “destroy and fry this paper” which leads to the nitpicking we keep seeing.

More drastically, unfortunately, we keep seeing on X that academics are kind of doxing their peers. How we ended up here? Why an author should be mortified for using AI in their writing? I believe we should be grateful to have this technology in hand now. I will keep using AI for writing my CS papers. But if you ask me if I am an academic in the field literature, humanities or I am a writer/story teller, then I am 100% on the same page with you. I will ban AI for myself for writing for a lifetime to maintain genuine human footprint.

Certainly, I may be biased in my view and I acknowledge that. At the end of the day, we are a drop in the ocean. But there’s also one great piece by Mawlānā that I adore:

«تو قطره‌ای در اقیانوس نیستی، تو تمام اقیانوس در یک قطره‌ای.»

Which translates to: “You are not a drop in the ocean. You are the entire ocean in a drop.”

So we all are part of the same Ocean 🌊 :-)

I am asking you esteemed researchers, members of the community, please hear my voice, share and support, which is stemming from more than 14 years of research in the field. We need to do something about peer review for these venues. it’s already too late …

Best wishes, Authors of submission 10206

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.