acceptodds
Under review as a conference paper at ICLR 2027

Ikigai is All You Need: Non-Gradient Referent-Ledger Construction Escapes the Bounded-Window Ceiling

Abstract

Gradient-trained language models pay a fixed, growing compute cost to exploit more data: acquiring new knowledge means reprocessing the entire corpus at full training cost. A constructive text generator built with no neural network, no gradient descent, and no transformer does not share this problem. Its acquisition cost grows sub-linearly with corpus size (wall-clock time   N^0.51 across a 122x corpus sweep, process memory flat at about 50 MB for its language model), and, reversing every gradient-scaling law we are aware of, its per-sentence generation speed and reliability actually improve as the underlying corpus grows, by 3.4-4.8x across independently replicated seeds, simply because richer real evidence makes its constraint-satisfaction search easier rather than harder. The mechanism also closes a structural gap that has limited non-gradient generation: any admissibility predicate evaluated over a bounded context window is provably capped at a regular language (Theorem A), and every surface-marker escape mechanism we constructed and tested needed to reintroduce a fixed convention just to stay decodable (Finding B). Unbounded, non-surface generator memory, an exact-identity referent ledger, escapes both, with structural complexity that keeps growing with the tracked-referent pool rather than saturating. At equal training-corpus size, two independent blind human raters judged this mechanism more natural than a converged gradient-trained transformer baseline on the exact matched corpus at matched diversity (2.31 vs 1.94 of 3, p=0.011), and it matches that baseline on two standard diversity and copying measures under the same matched-corpus condition (Distinct-2: 0.766 vs 0.773; copied 7-grams: 0% vs 0.7%) — all with zero backpropagation, zero GPU requirement, and flat RAM regardless of corpus size for its constant-work language model. One axis it does not yet win is perplexity under an external pretrained model, which by construction rewards statistically generic text; we report it rather than omit it. The same clustering mechanism can also directly organize a pretrained transformer's own embedding weights, the literal parameters rather than a distillation, into categories that tie or exceed our from-scratch results. It does not yet match transformer-scale generation unconditionally, and we present it as a first, concrete step in an active research program toward that goal rather than a finished system.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
non-gradient learning, text generation, formal language theory, constraint satisfaction, regular languages, natural language generation, symbolic AI, computational complexity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eFpEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates text generation without gradient-based training by combining a count-based language model, a constrained generator ( G* ), and an external referent ledger that enables previously introduced words to recur at long distances. It presents a formal analysis of bounded-window admissibility predicates and evaluates the proposed system against a small Transformer trained on the same Project Gutenberg corpus. The authors report improved likelihood over simpler count models, favorable sentence-level naturalness ratings in a small human study at approximately matched diversity, and reliable long-distance lexical recurrence. They also acknowledge a remaining coherence gap and explore several unsuccessful approaches to reducing it.

Critical strengths

S1. The paper honestly reports extensive ablations and negative results, providing useful evidence about the strengths and limitations of the proposed approach. The authors also acknowledge the remaining coherence gap and disclose corrections to earlier experiments, which improves transparency.

Critical weaknesses

W1. The writing and experimental presentation require substantial revision. The paper is terribly formatted and written, with significant signs of LLM-generation, consistent with the disclosure in Appendix A. This makes it difficult to identify and follow the core contribution. The abstract jumps between computational efficiency, formal-language theory, human evaluation, and embedding analysis without a clear narrative, and fails to provide a clear reason why a non-gradient-trained language model (as I quote from L.013) is better. The main text uses slogan-like phrasing, such as In one sentence, quoted from L.051, and An honest limit, quoted from L.104, which has a sign of insufficiently edited AI-generated paraphrase. These expressions should be replaced with precise descriptions of the findings and limitations. Figure 1 needs clearer spacing, and authors should avoid clipping of the illustrations. The experimental section is also difficult to follow. While the individual components are described, the presentation does not clearly distinguish standalone baselines, cumulative model improvements, and generator ablations. Relevant comparisons are scattered across Tables 1–3 and the surrounding text, making it difficult to isolate each component’s contribution. The authors should reorganize the experiments around explicit research questions, consistently identify the configuration evaluated in each row, and clarify which comparisons control for decoding strategy, diversity, and copying restrictions.

W2. The methodological novelty and practical utility of G* need stronger justification. For novelty, the system combines attested n-gram continuation sampling, copying restrictions, co-occurrence-based filtering, and candidate selection, drawing on established ideas in corpus-based generation and lexical constraints (Langkilde and Knight, 1998; Hokamp and Liu, 2017). Table 2 provides useful ablations demonstrating the benefits of the added states and selection mechanisms. However, it remains unclear which aspects of their integration constitute the principal methodological contribution. In particular, the recurrence experiment compares the ledger-equipped system against baselines without equivalent memory and target-word constraints. A comparison controlling for these capabilities would help distinguish the contribution of the proposed anchoring mechanism from the general benefit of explicitly storing and reintroducing a target word. For practical utility, the reported coherence errors raise concerns about its suitability for applications, which require meaningful and consistent text, and authors admit that transformer without greedy decoding do perform better than their system. Sentence-level naturalness ratings and exact target-word recurrence do not establish usefulness for a concrete generation task, and no downstream evaluation is provided. If the intended advantage is affordable local generation, the authors should compare quality and resource consumption against compact neural models running through optimized frameworks such as llama.cpp.It remains unclear whether the computational savings compensate for the limitations in text quality.

Refeneces are provided in the paper's Reference section.

Questions for authors

Q1. Can authors do any downstream tasks to prove the quality of generated tasks such as LAMBADA(Paperno et al. 2016) or ARC-Easy(Clark et al. 2018)?

Q2. Can authors carefully explain how Human rating (L.344) is done? Besides, I think that two raters seem quite few. Can the authors clarify the human evaluation procedure, including rater recruitment, annotation instructions, the scoring rubric, and sentence selection?

References

  • Paperno, D., et al. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context.
  • Clark, P., et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.
Reviewer_AuLUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a text-generation system based on corpus statistics and explicit memory, without neural networks or gradient descent. It combines Kneser–Ney smoothing, SVD-based similarity, word co-occurrence statistics, and EM-fitted latent states. Its generator, G*, samples from attested contexts while restricting copying. A referent ledger stores salient words introduced earlier and explicitly anchors their recurrence at long distances.

Compared with a 7.27M-parameter Transformer trained on the same Gutenberg corpus, the count language model closes 78% of the held-out NLL gap between its initial baseline and the Transformer. In a small human study at approximately matched diversity, G* receives higher naturalness ratings. The ledger also ensures target-word recurrence at distances of roughly 3,500 words, although structural coherence remains weaker than the Transformer’s. The theoretical analysis establishes the regular-language ceiling of finite-radius local predicates and explicitly clarifies that the ledger escapes fixed-window restrictions, not regularity itself.

Critical strengths

1. An interpretable non-neural system with component-level analysis.

  • The paper goes beyond assembling classical statistical methods by showing how individual components affect local prediction, sentence termination, compatibility with earlier words, and long-distance recurrence. Separating local evidence from external memory helps clarify the system’s capabilities and limitations.

2. Evaluation considers contamination and the quality–diversity trade-off.

  • The authors identify the advantage that corpus duplication gives count models and evaluate likelihood on test sentences without training-overlapping 7-grams. They also conduct human evaluation at approximately matched diversity and disclose coherence deficiencies and unsuccessful improvements. This provides stronger evidence than relying on perplexity alone.
Critical weaknesses

Overall, the individual results are interesting, but the proposed system’s advantage over existing alternatives remains unclear and requires end-to-end cost evaluation and comparisons with baselines under matched conditions.

1. The headline efficiency claims combine results from different systems.

  • The abstract’s N0.51N^{0.51} acquisition scaling and 3.4–4.8× generation speedup come from the earlier AC-3 generator, according to Appendix G. Human preference is demonstrated for the current G*, while approximately constant 50MB memory is reported for the disk-backed language model. The earlier generator’s peak memory actually increases from 190MB to 3.3GB. Combining these results risks implying that one current system demonstrates all these advantages simultaneously. A consistent end-to-end scaling evaluation of G*, including refitting time, memory, and disk costs, is needed.

2. The ledger demonstrates long-distance lexical recall, but not a distinctive expressivity advantage of the proposed approach.

  • The ledger explicitly forces a stored word into a sentence as an anchor, whereas the comparison systems are G* without the ledger and a Transformer with a 64-token context. The 100% versus 0% recurrence gap therefore largely reflects differences in access to external memory and target-word enforcement. Comparisons with a Transformer given the same ledger or lexical constraints would better isolate the proposed method’s additional value. Given the authors’ acknowledgment that the ledger is finite-state over a fixed vocabulary, the result is best interpreted as successful explicit word tracking beyond a local window, rather than evidence of non-regular expressivity or general discourse understanding.
Questions for authors
  1. Can the efficiency claims be demonstrated for the current G system end to end?
  • The acquisition scaling and generation speedup come from the earlier AC-3 generator, while the flat-memory result concerns the disk-backed language model. Could you report a unified scaling evaluation of G*, including preprocessing, refitting, generation, peak memory, and disk usage?
  1. Does the ledger provide an advantage under matched memory and constraint access?
  • Could you compare ledger-equipped G* with a Transformer given the same external memory and target-word constraints? This would clarify whether the recurrence advantage reflects the proposed generation mechanism or simply explicit storage and anchoring.
Reviewer_rk8dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores text generation without neural networks or gradient-based training. The proposed system generates sentences using word and phrase counts, with additional rules to guide topic consistency, sentence endings, and copying. An external ledger stores previously introduced words and brings them back into later sentences. On a Project Gutenberg corpus, the system approaches a small transformer’s generation quality and receives higher human naturalness ratings at roughly matched diversity. However, it remains worse at next-word prediction and coherence. The paper also studies the limits of fixed-window methods and discusses how external memory can extend their capabilities.

Critical strengths
  1. The paper explores an interesting alternative to neural text generation, showing how much can be achieved with corpus statistics and explicit memory alone.

  2. Training both systems on the same corpus is a useful experimental choice, making the comparison less dependent on differences in training data.

  3. The blind human evaluation provides evidence beyond model-based scores. At roughly matched diversity, raters preferred the proposed system’s sentences to the small transformer’s outputs.

  4. The component ablations help explain which design choices improve sentence endings, fluency, and long-distance word recurrence.

  5. The authors openly report failed approaches and remaining weaknesses, particularly the coherence gap. This makes the study informative even where the method falls short.

  6. The CPU-based implementation and explicit accounting of disk storage and process memory provide useful evidence about the practical costs of count-based language modeling.

Critical weaknesses
  1. The long-distance recurrence experiment gives the proposed system access to a stored target word and explicitly anchors generation on it, while the transformer has only a 64-token context. The 100% versus 0% result therefore does not isolate the value of the proposed mechanism. A transformer given the same target word or external memory is needed to establish a meaningful advantage beyond forced word inclusion.

  2. The theoretical results do not fully support the explanation of the system’s capabilities and limits. Corollary A.1 concerns ordered sequence copying, whereas the ledger stores an unordered set of words. Likewise, the failure of several marker-free constructions does not establish that clause-level coherence requires a stack or fixed surface conventions. These gaps weaken the paper’s central connection between its theory and empirical findings.

Questions for authors
  1. Can you repeat the recurrence experiment with the transformer given the same stored target word, through prompting or constrained decoding? Report both recurrence and sentence quality to separate the benefit of external memory from the proposed generation mechanism.

  2. Does the ledger preserve the meaning and associated facts of recurring words across a passage? An evaluation beyond exact word inclusion is needed to establish entity consistency.

  3. How does human preference change across transformer decoding settings? Evaluating several settings would clarify whether the advantage holds beyond the selected matched-diversity comparison.

  4. Do the scaling results hold for the current G* system? Report acquisition time, peak memory, disk usage, and generation quality across corpus sizes, since the headline scaling results come from the earlier generator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.