- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces N²-MARS, a native non-monotonic autoregressive decoding framework that equips pretrained LMs with a parameterized rollback primitive ⟨UNDO⟩N⟨/UNDO⟩ to delete the last N visible tokens and continue generation within a single pass. The authors construct a large on-policy rollback corpus from model-generated completions over open-domain texts judged by an LLM, train with a hybrid continued pretraining (HyCPT) objective that masks errors while supervising rollback and repair, and refine rollback triggering via GRPO with outcome- and help-based rewards. Experiments across 12 model/backbone settings show improved long-context generation quality and token efficiency, along with gains on math/planning, suggesting that rollback learned from open-domain text transfers to long-form generation.
Critical strengths
1.Introduces a simple but powerful parameterized rollback command that deletes multi-token spans in one shot, learned as a native generation primitive rather than a task-specific mechanism.
2.Proposes HyCPT with source-specific masking to condition on erroneous histories without reinforcing them, unifying continued pretraining and masked SFT in one stage.
3.Designs a judged, on-policy rollback corpus that captures real model errors with detection lag (placing ⟨UNDO⟩ at the end of a rejected span), aligning supervision with how errors surface in practice.
Critical weaknesses
1.Lack of direct comparison with closely related non-monotonic baselines. The paper does not compare against N-MARS (which performs repeated single-token erasures) or more recent in-stream edit primitives (e.g., Stream of Revision) in non-code settings. This omission makes it difficult to isolate the specific advantage of a parameterized, single-command rollback over repeated atomic operations. It is recommended to add a controlled baseline emulating N single-token erasures to report both generation quality and token/latency costs.
2.Reliance on LLM-as-Judge for both corpus construction and reinforcement learning rewards introduces risks of reward hacking and circularity. While a human agreement study is included, broader robustness checks are limited. For example, the authors should verify sensitivity to judge drift: if a different judge model (e.g., replacing Qwen3.8-27B with Claude Opus) is used to evaluate the final policy during the RL stage, does the model performance degrade significantly?
3.Statistical significance and variance across multiple random seeds are not reported for the main tables. Given the breadth of the experimental results, providing confidence intervals or standard deviations would significantly strengthen the reliability of the claims.
Questions for authors
1.How is the KV-cache handled at inference when executing the ⟨UNDO⟩N⟨/UNDO⟩ command? Does the system trigger the physical release and reallocation of the KV-cache (achieving constant-time cache rollback), or does the model merely truncate the sequence logically while retaining redundant computation of historical tokens in the underlying computation graph? Please clarify the decoding and runtime mechanism, and provide specific inference latency or throughput overhead data for large values of N.
2.What safeguards are in place to prevent malformed or degenerate rollback behavior (e.g., N being greater than the visible length, nested commands, or repeated rollbacks causing infinite loops)? Although the paper mentions penalizing malformed commands in the RL reward function, are there hard constraints (such as logit masking) deployed during the inference stage? Please provide failure-rate statistics and describe any beam- or entropy-based mitigation strategies.
3.Could you include a controlled baseline experiment that emulates N-MARS by issuing N consecutive single-token erasure commands to delete the exact same span, and report both the generation quality and the token/latency cost? This would directly demonstrate the efficiency benefit of the proposed parameterized rollback mechanism.
2 comments
Q4: How is the KV cache handled on a rollback command: physical release or logical truncation? Give latency and throughput overhead for large N.
A4: By default, the decoder performs logical truncation. Specifically, N$^2$-MARS decodes the raw trajectory in a single pass, and the interpreter maps it to the visible tokens after each command, so the rejected span stays in the KV cache. Since a command deletes exactly the last N visible tokens, our decoding also natively supports physical release, evicting the N rejected KV entries and re-indexing positions. We compare these two methods in Appendix D.3 and find that logical truncation is the better default on both general and long-context generation benchmarks. Moreover, physical release evicts the rejected span, the very evidence that stops the model from regenerating the same error, so the model may repeat and roll back that error until the generation budget is reached.
We also conduct experiments to measure the latency and throughput in Section 3.6 of the revised PDF, and find that N$^2$-MARS decodes at the throughput of the base model with latency close to Base, and is faster than Best-of-N in 30 of 32 settings and faster than Self-Refine in 28 of 31. This validates the efficiency of our proposed method.
Q5: Safeguards against malformed or degenerate rollbacks (N larger than visible length, nesting, infinite loops); hard constraints like logit masking at inference; failure-rate statistics.
A5: Inference enforces the command grammar with interpreter rules rather than logit masking. A command whose N is zero, larger than the visible stack, or non-numeric, or that lacks the closing marker, is a hidden no-op logged as a rejected event, and nesting is impossible because an opening marker inside a payload ends the enclosing command. Rolled-back tokens count toward the maximum number of generated tokens, so a rollback cannot loop indefinitely.
In Appendix D.4 of the revised PDF, we log per-benchmark interpreter events and list the average per backbone in the following table.
| Average per backbone | Llama-3.2-3B | Llama-3.1-8B | Qwen3.5-4B | Qwen3.5-9B |
|---|---|---|---|---|
| Responses with a command (%) | 20.2 | 11.2 | 33.6 | 26.2 |
| Malformed, of those (%) | 10.7 | 12.4 | 13.8 | 8.2 |
| Raw tokens rolled back (%) | 6.5 | 3.9 | 13.2 | 11.6 |
From the table, we conclude that N$^2$-MARS issues commands selectively, malformed commands are a small minority, and almost all malformed commands have N larger than the stack.
Q6: Controlled experiment emulating N-MARS with N consecutive single-token erasures over the same span, reporting quality and token/latency cost.
A6: Please see A1 for generation quality and token cost against N-MARS (Appendix D.2) and A4 for latency (Section 3.6).
We thank the reviewer for the careful review and the constructive suggestions on baselines, judging robustness, and inference safeguards, and we have added the corresponding experiments to the revised PDF to alleviate your concerns.
A1: We thank the reviewer for this suggestion and have added N-MARS, together with Self-Refine and Best-of-N, on all general and long-context benchmarks in Appendix D.2 of the revised PDF. For Stream of Revision, we could not find public source code to reproduce it faithfully. Moreover, we clarify that N-MARS is task-specific: without our released training corpus, it cannot be trained as a native model, making the corpus itself a direct contribution of our work. For a fair comparison, we train N-MARS on our corpus, and list direct comparisons with N-MARS over long-context generation tasks in the following table.
N$^2$-MARS scores higher than N-MARS with fewer generated tokens on all three benchmarks for both backbones, showing the effectiveness and efficiency of N$^2$-MARS. The token saving comes from the parameterized command: one command removes a whole span that token-level erasure must delete token by token, so the advantage grows with output length. We will discuss the latency in A4. The full experimental results, including N-MARS, Self-Refine, and Best-of-N, over general and long-context benchmarks are in Appendix D.2 of the revised PDF.
A2: We thank the reviewer for this point. To test for drift, we sample the policy before RL (HyCPT) and after RL (N$^2$-MARS) on the same 1,000 corpus prefixes, score both with the training judge Qwen3.8-27B and two judges from other families, GPT-5.6 Sol and Claude Opus 5, and list the share of Wrong verdicts (%) in the following table.
According to the table, all three judges assign fewer Wrong verdicts to N$^2$-MARS than to HyCPT, and the two judges that play no role in training confirm the improvement measured by the training judge, showing that the gain from RL is not an artifact of the judge that provides the reward. The complete results over all four backbone models are provided in Appendix D.7 of the revised PDF.
A3: We thank the reviewer for this suggestion. We run each method three times with different seeds on both the general and long-context generation benchmarks, report every run together with the mean and standard deviation, and list the results of N$^2$-MARS and the two strongest baselines in the following table (mean ± std).
We observe that N$^2$-MARS has the highest mean in every column on both backbones with a small standard deviation, showing that the gains of N$^2$-MARS are stable across runs rather than an artifact of a single run. We refer the reviewer to Appendix D.8 of the revised PDF for the complete experimental results, including every run of baseline methods and N$^2$-MARS over all four backbone models on all employed benchmarks.