- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates text generation without gradient-based training by combining a count-based language model, a constrained generator ( G* ), and an external referent ledger that enables previously introduced words to recur at long distances. It presents a formal analysis of bounded-window admissibility predicates and evaluates the proposed system against a small Transformer trained on the same Project Gutenberg corpus. The authors report improved likelihood over simpler count models, favorable sentence-level naturalness ratings in a small human study at approximately matched diversity, and reliable long-distance lexical recurrence. They also acknowledge a remaining coherence gap and explore several unsuccessful approaches to reducing it.
Critical strengths
S1. The paper honestly reports extensive ablations and negative results, providing useful evidence about the strengths and limitations of the proposed approach. The authors also acknowledge the remaining coherence gap and disclose corrections to earlier experiments, which improves transparency.
Critical weaknesses
W1. The writing and experimental presentation require substantial revision. The paper is terribly formatted and written, with significant signs of LLM-generation, consistent with the disclosure in Appendix A. This makes it difficult to identify and follow the core contribution. The abstract jumps between computational efficiency, formal-language theory, human evaluation, and embedding analysis without a clear narrative, and fails to provide a clear reason why a non-gradient-trained language model (as I quote from L.013) is better. The main text uses slogan-like phrasing, such as In one sentence, quoted from L.051, and An honest limit, quoted from L.104, which has a sign of insufficiently edited AI-generated paraphrase. These expressions should be replaced with precise descriptions of the findings and limitations. Figure 1 needs clearer spacing, and authors should avoid clipping of the illustrations. The experimental section is also difficult to follow. While the individual components are described, the presentation does not clearly distinguish standalone baselines, cumulative model improvements, and generator ablations. Relevant comparisons are scattered across Tables 1–3 and the surrounding text, making it difficult to isolate each component’s contribution. The authors should reorganize the experiments around explicit research questions, consistently identify the configuration evaluated in each row, and clarify which comparisons control for decoding strategy, diversity, and copying restrictions.
W2. The methodological novelty and practical utility of G* need stronger justification. For novelty, the system combines attested n-gram continuation sampling, copying restrictions, co-occurrence-based filtering, and candidate selection, drawing on established ideas in corpus-based generation and lexical constraints (Langkilde and Knight, 1998; Hokamp and Liu, 2017). Table 2 provides useful ablations demonstrating the benefits of the added states and selection mechanisms. However, it remains unclear which aspects of their integration constitute the principal methodological contribution. In particular, the recurrence experiment compares the ledger-equipped system against baselines without equivalent memory and target-word constraints. A comparison controlling for these capabilities would help distinguish the contribution of the proposed anchoring mechanism from the general benefit of explicitly storing and reintroducing a target word. For practical utility, the reported coherence errors raise concerns about its suitability for applications, which require meaningful and consistent text, and authors admit that transformer without greedy decoding do perform better than their system. Sentence-level naturalness ratings and exact target-word recurrence do not establish usefulness for a concrete generation task, and no downstream evaluation is provided. If the intended advantage is affordable local generation, the authors should compare quality and resource consumption against compact neural models running through optimized frameworks such as llama.cpp.It remains unclear whether the computational savings compensate for the limitations in text quality.
Refeneces are provided in the paper's Reference section.
Questions for authors
Q1. Can authors do any downstream tasks to prove the quality of generated tasks such as LAMBADA(Paperno et al. 2016) or ARC-Easy(Clark et al. 2018)?
Q2. Can authors carefully explain how Human rating (L.344) is done? Besides, I think that two raters seem quite few. Can the authors clarify the human evaluation procedure, including rater recruitment, annotation instructions, the scoring rubric, and sentence selection?
References
- Paperno, D., et al. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context.
- Clark, P., et al. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.