acceptodds
Desk rejected at ICLR 2027

Native Non-Monotonic Autoregressive Sequence Modeling

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive models generate sequences monotonically, i.e., every sampled token, even if erroneous, becomes a permanent condition for all subsequent steps, and the cost grows with the length of the generation. Non-monotonic autoregressive models address this limitation with a learned erase token, while existing designs roll back one token per command and learn from task-specific augmented traces, so the capability neither scales to long-form generation nor transfers beyond the tasks that supervise it. To this end, we propose N-MARS, **N**ative **N**on-**M**onotonic **A**uto**R**egressive **S**equence models, which introduce a rollback command <UNDO></UNDO> for autoregressive sequence models to roll back tokens as a native generation primitive learned from open-domain text. In particular, we construct a 9.5B on-policy rollback corpus by using model completions of open-domain texts and further employ LLM-as-Judge to annotate the completions, forming the rollback trajectories that allow models to recognize and roll back their own errors. Afterward, we design hybrid continued pre-training (HyCPT), a single stage that unifies continued pre-training and masked supervised fine-tuning under source-specific loss masks, followed by reinforcement learning with outcome rewards that refines when the command fires. Extensive experiments on three long-context generation benchmarks and nine general benchmarks, across four backbones from 3B to 9B parameters in two model families, demonstrate the efficiency and effectiveness of N-MARS.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Autoregressive Models, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dhfkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces N²-MARS, a native non-monotonic autoregressive decoding framework that equips pretrained LMs with a parameterized rollback primitive ⟨UNDO⟩N⟨/UNDO⟩ to delete the last N visible tokens and continue generation within a single pass. The authors construct a large on-policy rollback corpus from model-generated completions over open-domain texts judged by an LLM, train with a hybrid continued pretraining (HyCPT) objective that masks errors while supervising rollback and repair, and refine rollback triggering via GRPO with outcome- and help-based rewards. Experiments across 12 model/backbone settings show improved long-context generation quality and token efficiency, along with gains on math/planning, suggesting that rollback learned from open-domain text transfers to long-form generation.

Critical strengths

1.Introduces a simple but powerful parameterized rollback command that deletes multi-token spans in one shot, learned as a native generation primitive rather than a task-specific mechanism.

2.Proposes HyCPT with source-specific masking to condition on erroneous histories without reinforcing them, unifying continued pretraining and masked SFT in one stage.

3.Designs a judged, on-policy rollback corpus that captures real model errors with detection lag (placing ⟨UNDO⟩ at the end of a rejected span), aligning supervision with how errors surface in practice.

Critical weaknesses

1.Lack of direct comparison with closely related non-monotonic baselines. The paper does not compare against N-MARS (which performs repeated single-token erasures) or more recent in-stream edit primitives (e.g., Stream of Revision) in non-code settings. This omission makes it difficult to isolate the specific advantage of a parameterized, single-command rollback over repeated atomic operations. It is recommended to add a controlled baseline emulating N single-token erasures to report both generation quality and token/latency costs.

2.Reliance on LLM-as-Judge for both corpus construction and reinforcement learning rewards introduces risks of reward hacking and circularity. While a human agreement study is included, broader robustness checks are limited. For example, the authors should verify sensitivity to judge drift: if a different judge model (e.g., replacing Qwen3.8-27B with Claude Opus) is used to evaluate the final policy during the RL stage, does the model performance degrade significantly?

3.Statistical significance and variance across multiple random seeds are not reported for the main tables. Given the breadth of the experimental results, providing confidence intervals or standard deviations would significantly strengthen the reliability of the claims.

Questions for authors

1.How is the KV-cache handled at inference when executing the ⟨UNDO⟩N⟨/UNDO⟩ command? Does the system trigger the physical release and reallocation of the KV-cache (achieving constant-time cache rollback), or does the model merely truncate the sequence logically while retaining redundant computation of historical tokens in the underlying computation graph? Please clarify the decoding and runtime mechanism, and provide specific inference latency or throughput overhead data for large values of N.

2.What safeguards are in place to prevent malformed or degenerate rollback behavior (e.g., N being greater than the visible length, nested commands, or repeated rollbacks causing infinite loops)? Although the paper mentions penalizing malformed commands in the RL reward function, are there hard constraints (such as logit masking) deployed during the inference stage? Please provide failure-rate statistics and describe any beam- or entropy-based mitigation strategies.

3.Could you include a controlled baseline experiment that emulates N-MARS by issuing N consecutive single-token erasure commands to delete the exact same span, and report both the generation quality and the token/latency cost? This would directly demonstrate the efficiency benefit of the proposed parameterized rollback mechanism.

2 comments

We thank the reviewer for the careful review and the constructive suggestions on baselines, judging robustness, and inference safeguards, and we have added the corresponding experiments to the revised PDF to alleviate your concerns.

Q1: No comparison with N-MARS or in-stream edit primitives such as Stream of Revision. Add a controlled baseline emulating N single-token erasures with quality and token/latency cost.

A1: We thank the reviewer for this suggestion and have added N-MARS, together with Self-Refine and Best-of-N, on all general and long-context benchmarks in Appendix D.2 of the revised PDF. For Stream of Revision, we could not find public source code to reproduce it faithfully. Moreover, we clarify that N-MARS is task-specific: without our released training corpus, it cannot be trained as a native model, making the corpus itself a direct contribution of our work. For a fair comparison, we train N-MARS on our corpus, and list direct comparisons with N-MARS over long-context generation tasks in the following table.

BackboneMethodLongProcTokLongGenBenchTokLongBench-WriteTok
Llama-3.1-8BN-MARS5.21.1220.13.1851.20.91
Llama-3.1-8BN$^2$-MARS6.20.8422.42.4553.30.69
Qwen3.5-9BN-MARS15.32.2947.41.3686.81.29
Qwen3.5-9BN$^2$-MARS18.21.6750.61.0388.50.97

N$^2$-MARS scores higher than N-MARS with fewer generated tokens on all three benchmarks for both backbones, showing the effectiveness and efficiency of N$^2$-MARS. The token saving comes from the parameterized command: one command removes a whole span that token-level erasure must delete token by token, so the advantage grows with output length. We will discuss the latency in A4. The full experimental results, including N-MARS, Self-Refine, and Best-of-N, over general and long-context benchmarks are in Appendix D.2 of the revised PDF.

Q2: LLM-as-judge for both corpus and RL rewards risks reward hacking and circularity; test judge drift by evaluating the final policy with a different judge.

A2: We thank the reviewer for this point. To test for drift, we sample the policy before RL (HyCPT) and after RL (N$^2$-MARS) on the same 1,000 corpus prefixes, score both with the training judge Qwen3.8-27B and two judges from other families, GPT-5.6 Sol and Claude Opus 5, and list the share of Wrong verdicts (%) in the following table.

JudgePolicyLlama-3.1-8BQwen3.5-9B
Qwen3.8-27BHyCPT34.626.6
Qwen3.8-27BN$^2$-MARS31.923.1
GPT-5.6 SolHyCPT39.431.7
GPT-5.6 SolN$^2$-MARS37.826.1
Claude Opus 5HyCPT40.730.5
Claude Opus 5N$^2$-MARS36.625.0

According to the table, all three judges assign fewer Wrong verdicts to N$^2$-MARS than to HyCPT, and the two judges that play no role in training confirm the improvement measured by the training judge, showing that the gain from RL is not an artifact of the judge that provides the reward. The complete results over all four backbone models are provided in Appendix D.7 of the revised PDF.

Q3: No statistical significance, seeds or confidence intervals in the main tables.

A3: We thank the reviewer for this suggestion. We run each method three times with different seeds on both the general and long-context generation benchmarks, report every run together with the mean and standard deviation, and list the results of N$^2$-MARS and the two strongest baselines in the following table (mean ± std).

BackboneMethodIFEvalMATHLongProcLongGenBenchLongBench-Write
Llama-3.1-8BBest-of-N17.3 ± 1.220.9 ± 1.52.8 ± 0.616.8 ± 5.136.0 ± 1.8
Llama-3.1-8BN-MARS21.3 ± 0.425.1 ± 0.34.9 ± 0.318.2 ± 2.752.0 ± 0.7
Llama-3.1-8BN$^2$-MARS27.3 ± 0.527.6 ± 0.56.0 ± 0.320.7 ± 2.652.3 ± 1.2
Qwen3.5-9BBest-of-N49.8 ± 0.358.6 ± 0.52.2 ± 0.546.5 ± 3.684.8 ± 0.9
Qwen3.5-9BN-MARS57.0 ± 0.460.0 ± 0.514.8 ± 0.650.3 ± 2.986.4 ± 0.5
Qwen3.5-9BN$^2$-MARS59.0 ± 0.461.3 ± 0.317.5 ± 2.054.9 ± 3.787.5 ± 1.4

We observe that N$^2$-MARS has the highest mean in every column on both backbones with a small standard deviation, showing that the gains of N$^2$-MARS are stable across runs rather than an artifact of a single run. We refer the reviewer to Appendix D.8 of the revised PDF for the complete experimental results, including every run of baseline methods and N$^2$-MARS over all four backbone models on all employed benchmarks.

Q4: How is the KV cache handled on a rollback command: physical release or logical truncation? Give latency and throughput overhead for large N.

A4: By default, the decoder performs logical truncation. Specifically, N$^2$-MARS decodes the raw trajectory in a single pass, and the interpreter maps it to the visible tokens after each command, so the rejected span stays in the KV cache. Since a command deletes exactly the last N visible tokens, our decoding also natively supports physical release, evicting the N rejected KV entries and re-indexing positions. We compare these two methods in Appendix D.3 and find that logical truncation is the better default on both general and long-context generation benchmarks. Moreover, physical release evicts the rejected span, the very evidence that stops the model from regenerating the same error, so the model may repeat and roll back that error until the generation budget is reached.

We also conduct experiments to measure the latency and throughput in Section 3.6 of the revised PDF, and find that N$^2$-MARS decodes at the throughput of the base model with latency close to Base, and is faster than Best-of-N in 30 of 32 settings and faster than Self-Refine in 28 of 31. This validates the efficiency of our proposed method.

Q5: Safeguards against malformed or degenerate rollbacks (N larger than visible length, nesting, infinite loops); hard constraints like logit masking at inference; failure-rate statistics.

A5: Inference enforces the command grammar with interpreter rules rather than logit masking. A command whose N is zero, larger than the visible stack, or non-numeric, or that lacks the closing marker, is a hidden no-op logged as a rejected event, and nesting is impossible because an opening marker inside a payload ends the enclosing command. Rolled-back tokens count toward the maximum number of generated tokens, so a rollback cannot loop indefinitely.

In Appendix D.4 of the revised PDF, we log per-benchmark interpreter events and list the average per backbone in the following table.

Average per backboneLlama-3.2-3BLlama-3.1-8BQwen3.5-4BQwen3.5-9B
Responses with a command (%)20.211.233.626.2
Malformed, of those (%)10.712.413.88.2
Raw tokens rolled back (%)6.53.913.211.6

From the table, we conclude that N$^2$-MARS issues commands selectively, malformed commands are a small minority, and almost all malformed commands have N larger than the stack.

Q6: Controlled experiment emulating N-MARS with N consecutive single-token erasures over the same span, reporting quality and token/latency cost.

A6: Please see A1 for generation quality and token cost against N-MARS (Appendix D.2) and A4 for latency (Section 3.6).

Reviewer_QoMwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper teaches a language model to take back part of what it has written. It adds a command that deletes the last N tokens and lets the model continue with a rewrite in the same pass. Unlike earlier undo methods, one command removes a whole span, and training uses general text rather than a single task.

The training data comes from the model’s own mistakes. The model continues real documents, and an LLM judge marks each continuation as wrong or acceptable. A wrong continuation is followed by a command that deletes all of it, and then by the document’s real text as the repair. An acceptable continuation is kept without a command. The command is placed at the end of a wrong passage, because errors often become visible only after the model has written past them.

The model is first trained on these examples mixed with ordinary text and instruction data. It is then refined with reinforcement learning. Experiments on four models report gains on long-form generation and on general benchmarks, along with fewer generated tokens than methods that rewrite full drafts.

Critical strengths
  • Letting the model delete a whole span with one command, and then keep writing in the same pass, is a simple and useful idea. I expect it to draw interest.
  • The paper is clearly written and easy to follow.
Critical weaknesses
  • Undo and backtracking tokens already exist, including ones that remove a chosen number of tokens. The main new piece is training on general text instead of a single task. The novelty is therefore modest.

  • I am not convinced the gains come from rollback. On GSM8K, Llama-3.1-8B improves from 53.2% to 70.7%, yet it uses the command in only 0.5% of its responses. More than half of this gain comes from the RL stage. The trained model also gets instruction data and RL that the base model does not. No model in the paper receives the same instruction data and RL without the undo command.

  • The labels may not mark real errors. The judge labels most continuations as wrong, including almost all continuations of long books and reports. If many of these labels reflect differences from the reference rather than errors, the model learns to replace its text with human text rather than to fix mistakes.

Questions for authors
  • How much of the gain remains if the model receives the same data and reinforcement learning without the undo command? How does the method compare with the earlier single-token undo method?
  • On the benchmarks with the largest gains, how do responses that use the command compare with responses that do not?
  • How balanced is the human-annotated set used to validate the judge? How often does the judge mark acceptable continuations as wrong?
  • Does the model distinguish its own text from human text? For example, how often does it roll back correct model-written text, or keep human-written text that contains an error?
4 comments

We appreciate the reviewer's positive assessment of the idea and the writing, and we address each question below with new ablations and analyses.

Q1: Novelty is modest, and the new piece is only training on general text.

A1: We would like to emphasize that learning rollback as a native ability of existing sequence models for general-purpose tasks is the central problem of this work, and as discussed in Related Work (Section 4), prior backtracking methods are all task-specific and leave open for general tasks. Moving rollback to general text raises three questions that these works fail to alleviate, and our paper answers each of them: (i) Where do the errors come from? We sample the model's own completions over seven open-domain sources and judge them, forming a 9.5B on-policy rollback corpus (Section 2.1). (ii) Where does the command go, and how far does it reach? We place the command at the end of the rejected completion, where the error has become evident, and spell its span with the existing digit tokens, so one command removes a span of any length. (iii) How is it trained without reproducing the model's errors or degrading the backbone? HyCPT unifies continued pre-training and mSFT under source-specific loss masks (Section 2.2), and RL rewards a rollback only when it turns a wrong output into a correct one (Section 2.3). Moreover, our experiments show that the gain comes from these designs and the data together: N$^2$-MARS outperforms both N-MARS trained on our corpus and CPT + RL without the command on most benchmarks across all four backbones (see A4 and A2).

We would like to clarify our contributions: (i) N$^2$-MARS, the first method that learns multi-token, single-pass rollback of visible tokens from open-domain text to teach rollback as a native ability; (ii) a 9.5B judged on-policy rollback corpus built from the model's own completions of open-domain documents, which supplies rollback supervision without any task-specific data; (iii) HyCPT and an outcome-based reward design, where HyCPT installs the command under source-specific loss masks so the model conditions on its errors without learning to reproduce them. Moreover, the reward pays a bonus only for a verified wrong-to-right rollback and penalizes malformed commands, so the model learns when the command is worth firing.

Q2: Gains may not come from rollback; command used in 0.5% of GSM8K responses, over half the gain is from RL, and no model gets the same instruction data and RL without the command.

A2: We thank the reviewer for this point. To separate the contribution of the rollback command from that of the training pipeline, we apply the same RL stage to Base and CPT without the rollback command, with rewards {+1, -1} graded by the judge. We list the comparison over representative general and long-context benchmarks in the following table.

BackboneMethodIFEvalGSM8KHumanEvalLongProcLongBench-Write
Llama-3.1-8BBase + RL22.661.341.51.929.1
Llama-3.1-8BCPT + RL24.467.942.10.541.0
Llama-3.1-8BN$^2$-MARS27.770.742.72.453.3
Qwen3.5-9BBase + RL57.985.265.20.486.1
Qwen3.5-9BCPT + RL58.489.268.95.969.7
Qwen3.5-9BN$^2$-MARS59.389.770.26.888.5

N$^2$-MARS achieves the best result, ahead of Base + RL and CPT + RL, which receive the same data and RL stage, and the gap is largest on long-context generation, showing that RL alone does not account for N$ ^2$-MARS's gain. The full experimental results, including ablations over all four backbone models on all employed benchmarks, are provided in Tables 1 and 3 of the revised PDF.

Q3: Judge labels may mark divergence from the reference rather than real errors, especially on long books and reports, so the model may learn to replace its text with human text.

A3: We thank the reviewer for this substantive point. We would like to clarify that the judge is asked whether the completion is consistent with the prefix and internally plausible, rather than whether it matches the reference (judge prompt in Figure 9). We further validate the judge's verdicts against 500 human-annotated samples per corpus, drawn by stratified sampling on the judge's verdict with Wrong as the positive class, and list the results in the following table.

Corpus# Wrong# AcceptableAcc.Bal. acc.Prec.FPR
RollbackTraj-Qwen-3.528521592.693.599.60.5
RollbackTraj-Llama-331118993.093.196.06.3

Wrong verdicts are highly precise on both corpora (99.6% and 96.0%), so the rollback trajectories built from Wrong verdicts rarely teach the model to undo acceptable text. On long books and reports, the judged completions are the longest in the corpus, about 1.7K to 2.0K tokens on average against 0.6K to 1.0K on the other sources (Mean N in Table 8 of the revised PDF), and a longer completion from a small sampler is more likely to contain an error, which is consistent with their higher Wrong share. Appendix D.5 in the revised PDF reports the full error rates of the judge, including recall and the false negative rate on both corpora. We will discuss whether the model separates its own text from human text in A7.

Q4: How much gain remains with the same data and RL but no undo command, and how does it compare with the single-token undo method?

A4: For the same data and RL without the undo command, please see A2. For the single-token undo method, we report performance comparisons with N-MARS and list results over representative benchmarks in the following table. Note that N-MARS is task-specific: without our released training corpus, it cannot be trained as a single native checkpoint that performs well across all tasks. Therefore, we train N-MARS on our corpus to obtain a single checkpoint for a fair comparison with our model.

BackboneMethodIFEvalGSM8KMATHLongProc 0.5KLongGenBench STIC-1LongBench-Write S̄
Llama-3.1-8BN-MARS21.665.825.45.220.151.2
Llama-3.1-8BN$^2$-MARS27.770.727.86.222.453.3
Qwen3.5-9BN-MARS57.389.160.215.347.486.8
Qwen3.5-9BN$^2$-MARS59.389.761.618.250.688.5

According to the table, N$^2$-MARS outperforms N-MARS on all six benchmarks for both backbones, covering general and long-context tasks, demonstrating N$ ^2$-MARS's effectiveness. The advantage comes from the parameterized command: one command removes an entire erroneous span, whereas token-level erasure must delete it token by token.

We revised the PDF to include the full experimental results over all four backbone models on all general and long-context benchmarks, in Appendix D.2.

Q5: On the benchmarks with the largest gains, how do responses that use the command compare with those that do not?

A5: We thank the reviewer for this question. We first split the test instances of each benchmark by whether N$^2$-MARS triggers the command, and find that the command is triggered on instances that are harder for the backbone, and that the gain of N$^2$-MARS over the base model concentrates on these instances (Appendix D.1). To further isolate the effect of the command itself, we decode the same weights with the rollback command disabled, and list the comparison over representative benchmarks in the following table.

BackboneCommandIFEvalHumanEvalLongGenBenchLongGenBench
Llama-3.1-8BEnabled27.742.722.48.5
Llama-3.1-8BDisabled27.242.117.93.2
Qwen3.5-9BEnabled59.370.250.615.2
Qwen3.5-9BDisabled56.468.339.43.5

According to the table, disabling the command consistently lowers performance, especially on the backbones that fire the command most and on long outputs, showing that the gain of N$^2$-MARS comes from responses in which the command is actually executed.

The full experimental results, including the disabled-command decoding over all four backbone models on all benchmarks and the split of each benchmark by whether the command is triggered, are provided in Appendix D.1 of the revised PDF.

Q6: How balanced is the human-annotated judge set, and how often is an acceptable continuation marked wrong?

A6: The positive class is Wrong, and the 500 annotated samples per corpus are drawn by stratified sampling on the judge's verdict, since a random sample would be dominated by Wrong. The class split, balanced accuracy, and false positive rate are listed in the table of A3: balanced accuracy stays close to accuracy on both corpora, and the judge's errors are mostly missed Wrong verdicts rather than false alarms.

Q7: Does the model distinguish its own text from human text, e.g. rolling back correct model text or keeping erroneous human text?

A7: We test whether the rollback signal follows errors or the author of the text on the WikiBio hallucination set of SelfCheckGPT, which pairs 238 Wikipedia biographies with LLM-written passages about the same people and labels every sentence of these passages by hand as accurate, minor inaccurate, or major inaccurate. We score each sentence by the largest probability of the rollback command that N$^2$-MARS assigns while reading it, and list the AUROC (%) of separating major inaccurate from accurate model-written sentences (Errors) and accurate model-written from human-written sentences (Authorship) in the following table. Since a sentence that follows an error inherits its context, we also report the first sentence of each passage.

BackboneErrors (all)Authorship (all)Errors (first sentence)Authorship (first sentence)
Llama-3.1-8B65.157.181.124.0
Qwen3.5-9B67.954.575.239.8

According to the table, we find that inaccurate sentences score clearly above accurate ones, while accurate model-written sentences score close to human-written ones over all sentences and below them on the first sentence, showing that N$^2$-MARS does not roll back correct text for being model-written, and its rollback signal follows errors rather than the author of the text. Appendix D.6 in the revised PDF includes the full results and discussions for all four backbone models.

Thank you for the detailed response and the new experiments. The RL-only controls, the comparison with single-token undo on the same corpus, the disabled-command decoding, and the judge validation address my concerns about where the gains come from and whether the labels mark real errors. I have raised my soundness score accordingly.

My main concern remains novelty. The components come from prior work on the same problem, including N-MARS, and their combination follows naturally from the goal of training rollback on general text. The contribution is therefore mainly the large-scale data and its execution, so I keep my overall rating.

Reviewer_U3DKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a variable-length rollback command <UNDO>N</UNDO>. Training data comes from open-domain text: the model’s own continuations are judged by an LLM, and rejected ones become rollback trajectories. Training is HyCPT (source-specific loss masks) followed by GRPO, evaluated on 4 backbones across long-form generation and general benchmarks.

Critical strengths
  1. A single command rolls back a span of any length with only two new vocabulary entries, which is a clean design.
  2. Moving from task-specific to open-domain, on-policy supervision, and placing the command at the end of the erroneous span (detection lag), are well motivated.
Critical weaknesses
  1. Gains are not attributable to rollback. Llama-3.1-8B rolls back on only 0.5% of GSM8K responses, yet improves from 53.2 to 70.7. The RL prompt set is never specified, so reasoning gains may come from RL itself. A control with the same RL but no rollback command is missing.
  2. The most direct baseline, N-MARS, is absent. Table 1 also omits Self-Refine and Best-of-N.
  3. Rolled-back tokens remain in context. The model still conditions on the raw trajectory, so rollback only changes the visible output. This sits in tension with the stated motivation of stopping error propagation. KV-cache handling and actual latency are not discussed.
Questions for authors
  1. What results do HyCPT without rollback trajectories, and the same RL without the command, produce on Tables 1 and 2?
  2. What is the RL prompt set, and how much of it overlaps in domain with GSM8K, MATH and IFEval?
  3. If long-tranche clean examples are drawn from model-generated text, or rollback examples from human-written text, does rollback detection still hold?
  4. What are the class distribution, the positive class, and the balanced accuracy for the judge evaluation in Table 3?
3 comments

We thank the reviewer for the close reading of our experiments and the detailed questions, which helped us clarify the role of the rollback command and the training setup in the revision.

Q1: Gains are not attributable to rollback. The RL prompt set is never specified, so reasoning gains may come from RL itself. A control with the same RL but no rollback command is missing.

A1: To separate the contribution of the rollback command from that of RL, we apply the same RL stage to Base and CPT models without the rollback command, and list the comparison over representative general and long-context benchmarks in the following table.

BackboneMethodIFEvalGSM8KHumanEvalLongProcLongBench-Write
Llama-3.1-8BBase + RL22.661.341.51.929.1
Llama-3.1-8BCPT + RL24.467.942.10.541.0
Llama-3.1-8BN$^2$-MARS27.770.742.72.453.3
Qwen3.5-9BBase + RL57.985.265.20.486.1
Qwen3.5-9BCPT + RL58.489.268.95.969.7
Qwen3.5-9BN$^2$-MARS59.389.770.26.888.5

According to the table, N$^2$-MARS consistently outperforms both models at the same RL stage, and the gap is largest on long-context generation, showing that RL alone does not account for N$^2$-MARS's gain. We also analyze N$^2$-MARS with the rollback command disabled in Appendix D.1, and find that disabling the command lowers performance on nearly every benchmark, most on the backbones that fire it more frequently. The RL prompt set is obtained by randomly selecting 64K samples from the corpus by removing the overlap with benchmark train/val/test sets as discussed in Appendix B.5.

The full experimental results, including ablations over all four backbone models on all employed benchmarks, are provided in Tables 1 and 3. Moreover, Appendix D.1 analyzes the rollback command at inference time in the revised PDF.

Q2: The most direct baseline, N-MARS, is absent. Table 1 also omits Self-Refine and Best-of-N.

A2: We have conducted experiments on N-MARS, Self-Refine, and Best-of-N on the general and long-context benchmarks and report the results in the revised PDF. Note that N-MARS is task-specific: without our released training corpus, it cannot be trained as a native model, which highlights the corpus itself as a direct contribution of our work. For a fair comparison, we train N-MARS on our corpus and list the comparisons over representative benchmarks in the following table.

BackboneMethodIFEvalGSM8KMATHLongProcLongGenBenchLongBench-Write
Llama-3.1-8BSelf-Refine11.852.619.40.111.025.1
Llama-3.1-8BBest-of-N17.160.922.13.411.337.1
Llama-3.1-8BN-MARS21.665.825.45.220.151.2
Llama-3.1-8BN$^2$-MARS27.770.727.86.222.453.3
Qwen3.5-9BSelf-Refine49.283.250.92.044.883.1
Qwen3.5-9BBest-of-N50.089.559.02.744.785.6
Qwen3.5-9BN-MARS57.389.160.215.347.486.8
Qwen3.5-9BN$^2$-MARS59.389.761.618.250.688.5

We find that N$^2$-MARS has better performance than all baselines on both general and long-context benchmarks, demonstrating its effectiveness. Moreover, revision after generation (Self-Refine and Best-of-N) does not carry over to long outputs, whereas N$^2$-MARS corrects errors in place within a single pass.

The full experimental results, including all baselines over all four backbone models on all general and long-context benchmarks, are provided in Appendix D.2 of the revised PDF.

Q3: Rolled-back tokens stay in context, so rollback only changes the visible output and clashes with the error-propagation motivation. KV cache and latency are not discussed.

A3: We thank the reviewer for raising the inference mechanism. Keeping the rejected span in context is by design: the training trajectory conditions both the command and the repair on the rejected span, so the model detects an error from the evidence that follows it and writes a repair that does not repeat it. What rollback removes is commitment: the rejected span never enters the output, and the model treats it as a rejected draft rather than a premise for what follows. We have restated the mechanism after Equation 2.

Since a command deletes exactly the last N visible tokens, our decoding also natively supports physical release, evicting the N rejected KV entries and re-indexing positions. We compare these two methods in Appendix D.3 and find that logical truncation is the better default in both general and long-context generation benchmarks. Moreover, physical release evicts the rejected span, the very evidence that stops the model from regenerating the same error, so the model may repeat and roll back that error until the generation budget is reached.

We also conduct experiments to measure latency and throughput in Section 3.6 and find that N$^2$-MARS decodes at the base model's throughput, with latency close to Base, and is faster than Best-of-N in 30 of 32 settings and faster than Self-Refine in 28 of 31. This validates the efficiency of our proposed method.

Q4: Results of HyCPT without rollback trajectories and of the same RL without the command on Tables 1 and 2.

A4: We would like to clarify that the CPT rows in Tables 1 and 2 are indeed HyCPT without rollback trajectories, i.e., the same training samples with the rejected completion and the rollback command removed for samples in Dundo\mathcal{D}_\text{undo} (Appendix C), and we have made this clearer in the revision. We have added the full ablation results, including the same RL stage applied to Base and CPT over all four backbone models on all employed benchmarks, in Tables 1 and 3 of the revised PDF.

Q5: What is the RL prompt set and its domain overlap with GSM8K, MATH and IFEval?

A5: We thank the reviewer for raising this. The RL stage scores 64K samples on open-domain prompts sampled at random from the rollback corpora (RollbackTraj-Qwen-3.5 and RollbackTraj-Llama-3), and contains no benchmark data. As discussed in Appendix B.5, the corpora barely overlap with the test splits of all employed benchmarks, including the IFEval. Moreover, we measure the 13-gram overlap between the corpora and the official training splits of the benchmarks, and list it in the following table.

13-gram overlapGSM8K trainMATH trainAll six splits
# Examples7,4737,496384,877
RollbackTraj-Qwen-3.537580
RollbackTraj-Llama-337681

According to the table, only a negligible fraction of the training examples overlap with our corpora, and we remove these documents when sampling the RL prompts. We have updated Appendix B.5 of the revised PDF with the complete discussion.

Q6: Does rollback detection hold if long-tranche clean examples are model text, or rollback examples are human text?

A6: We thank the reviewer for this question. We test whether the rollback signal follows errors or the author of the text on the WikiBio hallucination set of SelfCheckGPT, which pairs 238 Wikipedia biographies with LLM-written passages about the same people and labels every sentence of these passages by hand as accurate, minor inaccurate, or major inaccurate. We score each sentence by the largest probability of the rollback command that N$^2$-MARS assigns while reading it, and list the AUROC (%) of separating major inaccurate from accurate model-written sentences (Errors) and accurate model-written from human-written sentences (Authorship) in the following table. Since a sentence that follows an error inherits its context, we also report the first sentence of each passage.

BackboneErrors (all)Authorship (all)Errors (first sentence)Authorship (first sentence)
Llama-3.1-8B65.157.181.124.0
Qwen3.5-9B67.954.575.239.8

Inaccurate sentences score clearly above accurate ones, while accurate model-written sentences score close to human-written ones overall and below them on the first sentence, showing that the rollback signal of N$^2$-MARS follows errors rather than the author of the text. The full experiments over all four backbone models are provided in Appendix D.6 of the revised PDF.

Q7: Class distribution, positive class, and balanced accuracy of the judge evaluation in Table 3.

A7: We draw the 500 annotated samples per corpus by stratified sampling on the verdict of the judge, since a random sample would be dominated by Wrong, and list the class split and error rates against human labels in the following table. Note that we treat class WRONG as the positive class.

Corpus# Wrong# AcceptableAcc.Bal. acc.
RollbackTraj-Qwen-3.528521592.693.5
RollbackTraj-Llama-331118993.093.1

From the table, we find that balanced accuracy stays close to accuracy on both corpora, so the agreement of the judge is not an artifact of the class split, and its errors are mostly missed Wrong verdicts rather than false alarms. The full error rates of the judge, including recall and the false negative rate on both corpora, are provided in Appendix D.5 of the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.