acceptodds
Under review as a conference paper at ICLR 2027

AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits

Abstract

Autoregressive decoding faces challenges for data that are not causally ordered, e.g., containing cross-references or future dependencies. Furthermore, once tokens are generated, they are fixed and cannot be modified or improved, unlike how humans can iteratively refine their thoughts. While diffusion LLMs partially address this issue, they do not naturally lend themselves to adding or removing arbitrary numbers of tokens to/from a pre-generated context. We design AutoIndexer as a generative model with a novel training and decoding mechanism that simulates a Chain of Edits, allowing the model to flexibly modify its prior outputs in hindsight via token insertions, substitutions, and deletions. The model learns to predict marker tokens for opening and closing edits, simultaneously moving a cursor to positions in the existing token sequence, analogous to a text editor. Training sequences are obtained by perturbing ground-truth labels with inverse chained edit operations, and the model learns to edit by undoing those inverses. The perturbation training allows AutoIndexer to be robust to erroneous context and adaptable to editing tasks such as code correction. We demonstrate that our method exhibits post-generation context-editing capabilities that most autoregressive and diffusion models lack without compromising performance on standard LLM benchmarks.

Primary area:
generative models
Keywords:
Large Language Models, Non-autoregressive decoding, discrete diffusion, Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rAu7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors argue that autoregressive language models can not directly modify previously generated tokens during decoding. To address this, they propose AutoIndexer, a flexible-order decoding method based on Chains of Edits. The method uses marker and cursor prediction heads to decide when and where to insert, delete, or replace tokens. It also introduces edit-aware positional encoding to support these operations while preserving KV caching. During training, the model learns to reverse randomly generated edit operations applied to clean text. The results show that AutoIndexer largely preserves the original performance and achieves competitive results against diffusion models. Additional experiments using synthetic text corruptions show that the model can detect and repair some errors through autonomous editing.

Critical strengths

The proposed decoding is interesting. It allows autoregressive models to insert, delete, and replace previously generated tokens. The edit-aware positional encoding might be a useful technical contribution that supports KV-cache reuse. The method can thus be adapted from pretrained models using LoRA.

The experimental results are generally positive. AutoIndexer largely preserves the performance of the original model on standard benchmarks. It performs competitively against diffusion models. The free-generation experiments also show that the model can independently initiate and perform edits.

Critical weaknesses

The novelty needs clearer positioning against prior text-editing work (Reviser by Diab, 2026). The proposed perturbation training and the learning to edit text through actions and constructed edit trajectories have been explored before [Gu et al., 2019; Shi et al., 2022]. The fused kernel and Chain-of-Edits RoPE are interesting ideas, but their individual benefits are not demonstrated.

The experiments do not sufficiently demonstrate the practical benefits of self-correction. The general benchmark results mainly show that AutoIndexer preserves the original performance, but not an improvement. The editing experiments use synthetic errors and lack direct comparisons with existing editing or self-refinement baselines. The autonomous editing results are also limited, especially for insertion and substitution. It remains unclear whether the method can effectively correct mistakes in its own generated outputs or improve performance on real tasks as stated in the introduction where autoregressive LLMs often fail.

The experimental scope and analysis are limited. The experiments mainly focus on Qwen3-8B, and the autonomous editing evaluation uses only 100 examples per condition. The paper also lacks ablation studies to isolate the contributions of its main components. It also does not report training cost or inference speed. Since the method adds editing actions, prediction heads, and specialized attention, these measurements are helpful.

Questions for authors
  • It is not accurate to state that autoregressive LLMs are not able to refine their outputs. They can reconsider and refine answers internally (e.g., in thoughts) before producing the final response to the user. In this sense, what practical advantage does AutoIndexer offer over these approaches?
  • How does its editing performance compare with existing editing or self-refinement methods on established benchmarks?
  • What is the individual contribution of Chain-of-Edits RoPE?
  • Table 5 shows limited success for insertion and substitution. What are the main reasons for these failures?
3 comments

We thank the reviewer for the feedback and for noting the interest of the decoding scheme, the positional encoding, and the KV-cache reuse.

TL;DR: (1) Novelty: Unlike prior editing methods, AutoIndexer supports learned cursor prediction, display-position updates, and exact KV-cache reuse across edits. (2) Practical benefit: Unlike internal reasoning, AutoIndexer directly revises previously generated output without regenerating the full text; comparisons with existing editing methods are future work. (3) Chain-of-Edits RoPE: Enables multiple edits in one training forward pass and avoids KV recomputation at inference. (4) Editing results: Exact-match metrics underestimate open-ended insertion/substitution repairs; we plan to expand evaluation to 10K samples. (5) Ablations and efficiency: training takes 2.6x longer per step, while inference has ~20% overhead without optimization.


Weakness 1: novelty and positioning. The paper positions the method against this work in Section 4 and Table 1. The closest prior work differs on the points that define our contribution. Gu et al. (2019) and the other insertion/deletion decoders are non-autoregressive, so token positions shift between steps and the KV cache cannot be reused. Reviser (Diab, 2026) predicts cursor locations in absolute token shifts (combining tokens that indicate shifts by 1, 2, 4, 8, 16, 32, etc.), and has to refresh the keys and values after each edit, while AutoIndexer localizes edits with a pointer network, reassigns positional encodings after each edit, and keeps an exact KV cache across multiple edits. As for the Shi et al., 2022 paper, it is ambiguous which paper the reviewer is referring to. Assuming it is the “Text Editing as Imitation Game“ paper, they also re-encode the edited text after each edit, whereas for AutoIndexer, edits happen inside one decoding stream with exact KV-cache reuse and updated positional encoding. Neither prior paper decouples generation order from display order of tokens. We will revise the related work section and elaborate on these distinctions.

On the components, the fused kernel is a training-efficiency device. Chain-of-Edits RoPE is not an optional add-on and is a core part of the method, since the standard RoPE factorization only works for static sequences (Section 2.3) and cannot express multi-edit position shifts in one packed forward pass.

Weakness 2 and question 2: evaluation of edit capabilities. Our model fine-tuning recipe for edits employs a generic diffusion-like denoising loop rather than fine-tuning directly on edit datasets. The point made in the paper is that the model attains new capabilities of editing its own context if required without sacrificing generative performance, unlike diffusion LLMs. To this end, we position our work as an alternative paradigm to diffusion training for LLMs, as opposed to as a frontier editing model. Having said this, additional experiments on edit benchmarks would strengthen our work. We consider your suggestion to be useful feedback and aim to include these comparisons for a camera-ready paper.

Weakness 2 and question 4: analysis of insertion and substitution cases. Successes for insertion and substitution are naturally more difficult to measure, since it is an open-ended generative task without a single correct solution (as opposed to deletion tasks where the solution is clear). The “exact repair“ measure is a useful signal that shows that the model can recover the original ground truth sequence with some probability, but this metric is not something that can be saturated to 100% accuracy, since that would disregard alternative correct possibilities. While it is possible to create other metrics such as using LLM-as-a-judge to assess the quality of the corrected sentence, that introduces additional sources of noise (such as the model choice for the LLM-as-a-judge, and the threshold at which the output is scored as correct), hence we reported quantifiable measures such as the exact repair and changes in Levenshtein distance.

...(Continued in the next post, post 1/2)

...(Continued from the previous post, post 2/2)

Weakness 3. The sample size of the rollout evaluation should not be an issue. We will rerun the evaluation with 10,000 samples to get a better precision. In terms of ablation studies, the main components (perturbation training, marker prediction head, cursor prediction head and Chain-of-edits RoPE) are only effective when combined together. We confirmed that without perturbation training and marker/cursor predictions turned off, the Chain-of-edits RoPE attention kernel behaves equivalently to a standard attention kernel.

In terms of computational efficiency, we ran profiling and obtained the below:

ModelPerturb probPadded lenEffective raw lenPeak alloc (GiB)Peak reserved (GiB)Step time (s)
Qwen3 LoRAn/a2048204819.921.30.543
AutoIndexer LoRA1.030722302 (+12.4%)20.023.81.411
AutoIndexer LoRA0.03072204919.420.40.978

Here, gradient checkpointing was on for all models, and the batch size was 1. As shown, the forward pass time is longer by a factor of 2.6x for AutoIndexer, but with this additional time it parallelizes 50 edits, so the compute time per edit has improved 20x. Due to perturbation training, the sequence length is slightly longer than the default Qwen3 training (the frequency of edits depends on the user choice, but in our setting the sequence length increased by around 12%).

In terms of inference time, AutoIndexer’s inference time is slightly longer than Qwen3 (around +20%). However, this is without any inference time optimization in place, because the CuTeDSL acceleration was only applied to the attention kernel at training time, and our priorities were not on inference time acceleration.

ModelBatchPrefill (s)Decode ms/stepTokens/sPeak above weights (GiB)
Qwen310.04140.624.60.12
AutoIndexer10.48447.520.30.22
Qwen380.13340.9193.80.97
AutoIndexer80.58653.4144.11.75

Question 1. We agree that reasoning models can refine an answer internally. That reasoning, however, refines a draft that is not part of the output, and the final answer must then be written out again. Tokens already emitted into the output cannot be changed in place, resulting in more token usage, which is the limitation we address. The advantage of our method is that minor corrections can be made without regenerating the entire text.

Question 3. Chain-of-Edits RoPE is what lets the model reassign token positions after each edit so that tokens are positioned in display order at each step, allowing parallel training over N number of context edits in one forward pass, and without having to recompute the keys and values at inference time. Without this mechanism, the model would have to reorder the tokens after each edit operation and perform N numbers of forward passes at training time, and have to recompute the keys and values over the entire sequence so far N number of times at inference time.

I appreciate the detailed response. The technical novelty is now much clearer, particularly around the position updates and KV-cache reuse. Action-based text editing is not really a novel concept, but the decoding architecture of AutoIndexer seems to be different from what has gone before. I also agree that an exact match is going to be a rather poor proxy for the quality of the insertion and substitution.

However my main issues remain. There are still no direct comparisons with existing editing methods. Although the paper evaluates standard benchmarks, it does not show whether editing improves performance on real tasks, such as correcting errors in generated code. Note that the introduction motivates the work with applications like code correction. My question was not on the sota performance, but on whether it actually offers meaningful advantages over existing approaches. The new efficiency results are useful but still do not establish that it is better. The larger evaluation that is planned may help, yet all I can go on at the moment is based on what I can see.

Overall, it does clarify a few things, but some issues are still there. So I intend to keep my current rating.

Reviewer_zZWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AutoIndexer, an autoregressive model that generates an edit stream rather than committing permanently to a left-to-right output. A marker head decides when to edit, pointer heads select the affected span, and a parser applies insertions, deletions, and substitutions. Training sequences are constructed by reversing synthetic edit operations. Edit-dependent rotary positions and attention masks separate generation order from display order while permitting reuse of cached states. The method is adapted from Qwen3-8B using LoRA and evaluated on standard language-model benchmarks and synthetic corruption-repair diagnostics.

Critical strengths

1.The integration of learned edit localization with an autoregressive generation history is a useful technical contribution. Sections 2.1–2.4 combine marker prediction, span pointers, edit-dependent positional encoding, and parsing into a mechanism that supports variable-length revisions without requiring externally specified edit spans. The generation/display-order distinction is a promising way to retain causal computation while allowing earlier output to be revised. 2.The experiments provide a credible initial demonstration of edit execution and substantial capability retention in the base setting. Table 2 includes a Qwen3 LoRA control trained on the same clean CPT data, with many scores close to AutoIndexer. Table 5 separately evaluates free generation, including abstention on clean inputs, and demonstrates exact repairs for all three edit types. These results make the proposal more substantive than an architectural sketch, while the reported failure cases help expose its current limitations.

Critical weaknesses

1.The evaluation establishes synthetic corruption repair more clearly than practically useful autonomous self-correction. The diagnostic uses 100 examples per category, one artificial corruption, and a 64-token rollout (Section 3; Appendix A.7). Table 5 reports exact repair rates of 10%, 53%, and 18% for insertion, deletion, and substitution, while 11% of clean examples also trigger an edit. These results demonstrate that the model can execute edits, but independently sampled noise differs from plausible mistakes generated by the model itself. Crucially, the standard language and code benchmark results do not establish whether editing improves final task correctness: the paper reports neither edit frequencies on these tasks nor a same-checkpoint comparison with editing enabled versus disabled. Consequently, the downstream benefit attributable to the editing mechanism remains unverified, independently of the model's ability to repair synthetic corruptions. 2.The practical advantage over a conventional autoregressive model using text-editing tools is not established. An autoregressive agent can already decide when and where to revise text using string-replacement or range-edit tools; for example, SWE-agent documents an editor taking start/end lines and replacement text (https://swe-agent.com/latest/usage/coding_challenges/). Thus, editable output and autonomous edit selection alone do not distinguish the method's practical capabilities from this alternative. The pointer heads, edit-aware attention, and cache treatment are meaningful architectural differences, and Section 5 discusses their distinction from tool calling. However, the paper provides no matched tool-mediated editing baseline or end-to-end measurements showing improved repair quality, reduced token/compute cost, lower latency, or lower memory use. It therefore remains unclear what practical benefit the additional architecture and training provide over a simpler tool-based editing interface. 3.Anomalies and inconsistencies in the quantitative reporting make the main empirical conclusions difficult to assess. Table 3 reports MBPP scores of 7.60 for matched Qwen3 LoRA, 64.00 for AutoIndexer, and 67.40 for original Qwen3. Although Section 3 and Appendix A.6 discuss SFT/few-shot regressions and mitigation, they do not resolve this exceptionally large gap between the matched models. Appendix A.6 also describes two few-shot-packed replay streams without corresponding entries or weights in Table A.2, leaving the final matched training recipe unclear. Table 5 reports insertion “Net improvement” of +11, although the exact-repair and improved categories already contain 10 + 12 cases. If these quantities use the same evaluation scope, this conflicts with the accompanying definition as a count of distance-reducing edits. These reporting issues require clarification to assess the matched-model comparison and the reported repair gains.

Questions for authors
  1. Incremental benefit of editing: On an existing evaluation such as HumanEval, can you report edit frequency and final task accuracy for the same AutoIndexer checkpoint with editing enabled versus disabled? A paired comparison, together with a small analysis of edits to model-generated mistakes, would help determine whether revision improves correctness rather than only repairing injected noise.
  2. Advantage over tool-mediated editing: What concrete benefit does AutoIndexer provide over the same backbone using a simple string-replacement or range-edit tool? Can you compare them on the existing repair diagnostic using identical drafts and task information, with clearly specified edit supervision, prompting, and comparable inference budgets? Please report repair quality and end-to-end token/compute cost or latency. Tool-based editors need not reproduce an entire document or even the old span when using range edits, and retaining an append-only interaction history does not inherently require discarding its existing KV cache. The comparison should isolate the benefit of learned pointers and edit-aware attention from edit training and interface overhead, rather than assume that tool use necessarily requires full re-encoding.
  3. Matched-control provenance: Is the Qwen3 LoRA MBPP value of 7.60 correct? Please explain why the gap persists despite the regression discussion and replay mitigation in Appendix A.6. Reconcile Table A.2 with the few-shot replay streams, specifying their datasets, splits, sampling weights, exclusion of evaluation examples, and whether both matched models used the identical final mixture. This would clarify how much weight to place on the capability-retention comparison.
  4. Repair metric: What exactly does “Net improvement” measure: the number of improved cases, improved minus worsened cases, or aggregate distance reduction? Please reconcile the +11 insertion and +31 substitution entries with the outcome counts and state whether distances are measured over the edited original window or include newly appended text.
  5. Meaning and verification of exact caching: Is exactness defined relative to recomputing the same edit history with its prefix-dependent positions and masks? Cached higher-layer states can retain information from tokens that are subsequently deleted, so this need not equal freshly encoding the current display sequence. Please make that scope explicit and provide a short invariance argument or numerical comparison of cached decoding, uncached decoding over the same history, and packed training. These are requests to clarify and validate the operator, not an assertion that the implementation is incorrect.
  6. Cost of the proposed mechanism: Can you report training/inference time and peak memory against matched Qwen3 on the existing workload, specifying raw generated tokens versus final displayed tokens? Appendix A.1's attention-FLOP discussion does not quantify routing/packing overhead, extra edit-generation steps, or the retained cache entries for deleted tokens. These measurements would establish the practical cost of the claimed cache-preserving design and complement the tool-mediated comparison above.
  7. Head architecture and reproducibility: Section 2.1 identifies the cursor head as a two-headed Q/K-only pointer network over final hidden states, while Figure 1 and Appendix A.2 describe the marker head's four-way decision. Could you provide the exact parameterization of both heads? In particular, is the marker head a single affine projection or an MLP, and what are the cursor Q/K projection dimensions, start/end parameter-sharing scheme, score scaling, and positional treatment? Please include any additional normalization/activation layers, initialization, and the total added parameter count.
4 comments

We thank the reviewer for the detailed review and for recognizing the integration of learned localization with the autoregressive history and the credible capability-retention results. We respectfully disagree that the evidence falls short of what the paper claims, and respond to each point below.

TL;DR: (1) Editing benefit: Our claim is autonomous editing without sacrificing LM performance, not that benchmark gains require edits; 62% of deletions and 47% of substitutions improve. We plan to expand evaluation to 10K samples. (2) Tool comparison: Unlike tool-based editing, AutoIndexer masks deleted tokens and updates display positions within attention; matched tool comparisons remain future work. (3) Reporting issues: We corrected the MBPP baseline (7.60 → 39.60), clarified that few-shot replay data were excluded, and will clarify the net-improvement metric. (4) Exact caching: Exactness refers to the same edit history, not re-encoding the final text. (5) Efficiency: Training takes 2.6× longer per step with edits; batch-1 decoding takes 47.5 vs. 40.6 ms/token, with further optimization possible. (6) Architecture: The marker head is a four-way classifier, and the cursor head is a two-headed pointer network; implementation details are provided in the supplementary code.


Weakness 1 and question 1: scope of the evidence and the role of editing. The paper makes two claims, each tested on its own axis (Section 3). First, standard capabilities are retained after conversion, which Tables 2 and 3 test against a matched LoRA control and diffusion LLMs. Second, the model can autonomously initiate and execute edits, which the diagnostic tests. Neither claim is that inference-time editing raises benchmark accuracy, and the HumanEval gain is offered only as an observation about the edit-training objective. An edits-enabled versus edits-disabled benchmark comparison would mostly measure how often a model needs to edit its own output. It is desirable for the model to get the answer right the first time, so that edits are unnecessary. What the edit capability provides is a way to recover from noise or mistakes in the context, and that is what the diagnostic measures.

The diagnostic uses no oracle. The model decides whether to edit (89 of 100 clean drafts left untouched, and 90 and 92 of 100 corrupted deletion and substitution drafts triggering an edit), localizes the span with its pointer heads (the correct span in 69 of 100 deletion and 66 of 100 substitution cases), and generates the replacement. The windows are drawn from held-out natural text and code (Appendix A.7), and we report the full outcome breakdown including failures and false positives (Table 5 and Appendix A.7.1). Controlled corruptions are what make repair measurable, since model-generated mistakes have no reference solution to score against.

The exact repair rates for insertion and substitution understate the results. These are open-ended tasks without a single correct solution, and exact repair counts only a verbatim match. By Levenshtein distance, 47 of 100 substitution cases and 62 of 100 deletion cases end strictly closer to the ground truth. LLM-as-a-judge scoring would add noise from the choice of judge and threshold. We will also rerun the rollout evaluation with 10,000 samples for better precision.

...(Continued in the next post, post 1/3)

...(Continued from the previous post, post 2/3)

Weakness 2 and questions 2 and 6: advantage over tool-mediated editing, and cost. The difference from a tool interface is in the mechanism, not the interface overhead. With a tool such as the SWE-agent editor, the model writes the edit location as text (line numbers or offsets) and the interaction history is append-only. Even if the KV cache is retained, the stale text stays in context at its original positions, still occupies position space, and is attended to by every later token. The model never sees the edited document in display order. AutoIndexer localizes the span with pointer heads, masks the deleted tokens, and reassigns positions after each edit, so later generation attends to the edited document with a KV cache that exactly matches the training computation (Table 1 and Section 5). Chain-of-Edits RoPE makes this possible. It allows parallel training over N context edits in one forward pass and inference without recomputing keys and values, whereas otherwise the model would need N forward passes in training and a full key and value recomputation per edit at inference.

The perturbation objective also builds edit-training sequences from any sequential dataset (Section 2.2), so no human edit histories, tool traces, or tool schema and agent harness are needed. We position our work as an alternative paradigm to diffusion training for LLMs, not as an agent scaffold or a frontier editing model, so a tool-calling baseline answers a different question from the one the paper asks. We aim to add comparisons on established edit benchmarks in the camera-ready version, and will expand the Section 5 discussion.

On cost, Appendix A.1 shows the fused kernel brings attention cost to about 1x that of conventional attention, down from 2x for a naive implementation. Perturbation training increases the sequence length (around 12% in our setting, with the edit frequency a user choice), and with flash attention memory scales linearly rather than quadratically. With perturbation training and the marker and cursor predictions turned off, the Chain-of-Edits RoPE kernel behaves equivalently to a standard attention kernel, so it computes the same attention as standard RoPE when no edits are present.

We ran profiling and obtained the below:

ModelPerturb probPadded lenEffective raw lenPeak alloc (GiB)Peak reserved (GiB)Step time (s)
Qwen3 LoRAn/a2048204819.921.30.543
AutoIndexer LoRA1.030722302 (+12.4%)20.023.81.411
AutoIndexer LoRA0.03072204919.420.40.978

Here, gradient checkpointing was on for all models, and the batch size was 1. As shown, the forward pass time is longer by a factor of 2.6x for AutoIndexer, but with this additional time it parallelizes 50 edits, so the compute time per edit has improved 20x.

In terms of inference time, AutoIndexer’s inference time is slightly longer than Qwen3 (around +20%). However, this is without any inference time optimization in place, because the CuTeDSL acceleration was only applied to the attention kernel at training time, and our priorities were not on inference time acceleration.

ModelBatchPrefill (s)Decode ms/stepTokens/sPeak above weights (GiB)
Qwen310.04140.624.60.12
AutoIndexer10.48447.520.30.22
Qwen380.13340.9193.80.97
AutoIndexer80.58653.4144.11.75

...(Continued in the next post, post 2/3)

...(Continued from the previous post, post 3/3)

Weakness 3 and questions 3 and 4: the quantitative reporting. The points raised concern how results are documented, and none of them changes the conclusions.

  • MBPP. The low score for the matched Qwen3 LoRA control was an evaluation artifact. That checkpoint yielded unstable results due to result formatting issues. We have rerun the LoRA training and evaluation and obtained 39.60, with the other baselines remaining similar. Under the same recipe AutoIndexer scores 64.00, well above the matched control and ahead of both diffusion models (37.60 and 58.00), and it matches or exceeds the control on most other benchmarks. The updated Table 3 is given below and will replace the original in the revision.
  • Table A.2. Table A.2 is correct. The two few-shot-packed replay sources mentioned in Appendix A.6 were removed from the final experiments to avoid overfitting to the benchmarks, and we will correct the Appendix A.6 text accordingly. Both matched models used the same data mixes and LoRA configuration (Tables A.1 and A.2, Appendix A.5).
  • Net improvement. The conclusions of the repair analysis rest on the outcome counts (exact repair, improved, neutral, worsened), which are consistent. Each category sums to 100, and the correct-span row equals the sum of its four span outcomes. Net improvement is a derived summary, and we will state its definition and window explicitly and report the underlying counts separately in the revision.
ModelQwen3 8BQwen3 8B LoRA*LLaDA 8B InstructDream 7B InstructAutoIndexer 8B LoRA*
Decoding strategyARARDiffusionDiffusionCoE (Chain of Edits)
Editable contextNoNoLimitedLimitedYes
---
General tasks
ARC-Challenge (0)90.6190.4486.2688.2390.61
MMLU-Pro (0)42.3847.9032.6544.3947.90
---
Mathematics & science
GPQA-Diamond (5)35.3529.8020.7135.8630.81
GSM8K (5)86.9682.4175.7480.5980.67
MATH500 (4)52.2048.0017.0045.8049.00
---
Code
HumanEval (0)83.5471.9545.1252.4475.61
MBPP (3)67.4039.6037.6058.0064.00

Question 5: meaning of exact caching. Yes, and this is by design. Exactness is defined relative to the same edit history with its prefix-dependent positions and masks, and it is not claimed to equal freshly encoding the final display sequence. A token's hidden states are computed while later-deleted tokens were still visible, and training matches this. The packed sequence is processed in one pass with per-segment positions and masks, so each query sees the keys that were visible at its own generation step (Section 2.3). Keys and values are cached unrotated and the rotation is applied at attention time, so cached and uncached decoding over the same history perform the same computation, and the fused kernel reconstructs the same joint softmax (Appendix A.1). The accelerated implementation was cross-checked against the manual implementation with unit tests, as stated in the paper. We will state this scope explicitly in Section 2.3.

Question 7: head architecture. The marker head is a four-way classifier over [NORMAL], [EDIT], [RETURN], and [EOS], and the cursor head is a two-headed pointer network over the final hidden states (Sections 2.1 and 2.5, Appendices A.2 and A.3). The full implementation and all hyperparameter configs are provided in the supplementary material and will be open-sourced, so the exact parameterization can be inspected directly.

Reviewer_zZWYOpenReview ↗

Response to rebuttal

Thank you for the detailed responses. The MBPP results for Qwen3 LoRA, the meaning of exact caching, and the model configuration are now clear to me, and I accept the authors’ explanations.

However, I still have two main concerns. First, I agree that the paper shows AutoIndexer can learn to edit while preserving its general capabilities. However, there is still not enough evidence that autonomous editing improves performance on practical tasks, such as code editing. Second, there is no direct experimental comparison showing whether AutoIndexer can match existing tool-based editing methods in both quality and efficiency.

I recognize the paper’s technical contribution, and the responses have addressed some of my concerns. However, the two concerns above remain, so I plan to keep my current rating. Thank you again for the responses.

Reviewer_VSrZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoIndexer equips a causal language model with learned marker and cursor heads to insert, delete, and substitute previously generated text. It separates generation order from display order, uses edit-aware rotary position encoding, and trains on synthetic chains of inverse edits. Experiments with Qwen3-8B LoRA models show broadly comparable performance to matched autoregressive baselines and demonstrate autonomous repairs of synthetically corrupted contexts.

Critical strengths
  1. The separation of generation and display order provides a coherent way to integrate variable-length editing into causal decoding. Learning both when and where to edit makes the approach more flexible than infilling with externally specified spans (Sections 2.1-2.4).

  2. The edit-aware positional encoding and reuse of unrotated cached keys and values are technically appealing. The implementation supports cache reuse under the proposed edit-history computation, while Appendix A.1 explains how the attention operation can be implemented without materializing dense attention matrices.

  3. The matched Qwen3 LoRA baselines help distinguish the effects of conversion from ordinary fine-tuning. Tables 2-3 show broadly preserved capabilities, including a HumanEval improvement from 51.2 to 59.2 in the base-model setting. The supplementary training, evaluation, and testing code also provides useful support for reproducibility.

Critical weaknesses
  1. Evidence for practical self-correction remains limited to small synthetic diagnostics with 100 examples per category. Table 5 reports exact repair rates of only 10% for insertion and 18% for substitution, with edits on 11% of clean contexts. These results establish an initial editing capability, but do not yet demonstrate reliable correction of naturally occurring errors or sustained multi-edit refinement.

  2. The contribution of active editing and its computational cost are not sufficiently isolated. There is no reported edit-disabled ablation on the standard benchmarks or end-to-end latency and memory comparison. Since discarded tokens remain in the cache and edits introduce additional decoding steps, cache reuse alone does not establish a favorable quality-efficiency tradeoff. Relatedly, in Table 3 the matched Qwen3 8B LoRA baseline scores only 7.60 on MBPP, versus 64.00 for AutoIndexer and 67.40 for the original Qwen3 8B, which suggests a failed run or evaluation issue for this control and weakens the matched comparison in the instruction-tuned setting.

Questions for authors
  1. Do the gains persist on naturally occurring model errors, and how does repair quality compare with a matched autoregressive model prompted to revise the same drafts under a comparable generation budget?

  2. How do benchmark accuracy, latency, and peak memory change when editing is disabled versus enabled, and as the number of edits increases? In particular, does the HumanEval gain require edits at inference time? Could the authors also explain the 7.60 MBPP score of the matched Qwen3 8B LoRA baseline in Table 3?

3 comments

We thank the reviewer for the positive assessment, in particular for recognizing the generation/display-order decoupling, the edit-aware positional encoding, and the reproducibility materials.

TL;DR: (1) Editing capability: AutoIndexer autonomously repairs 53% of deletions and improves 47% of substitutions, while leaving 89% of clean drafts untouched; we plan to expand evaluation to 10K samples. (2) Editing ablation: Our claim is that editing can be learned without sacrificing standard LM performance, not that benchmark gains require inference-time edits. (3) Efficiency: Current inference has ~20% overhead, with further optimization possible. (4) MBPP anomaly: We reran the Qwen3 LoRA baseline, correcting its score from 7.60 to 39.60 without changing our conclusions.


Weakness 1 and question 1: scope of the editing evidence. The paper's claim is that the edit capability can be introduced into a pretrained autoregressive model without loss of its broad capabilities, and that the resulting model initiates and executes edits on its own. The diagnostic is built to test exactly that, and it is a demanding one: the model must decide whether to edit with no oracle (89 of 100 clean drafts correctly left untouched), localize the span with its pointer heads, and generate the replacement. Under free generation it exactly repairs 53% of deletions, and 62% of deletion cases and 47% of substitution cases end with a strictly smaller distance to the ground truth.

Successes for insertion and substitution are naturally more difficult to measure, since it is an open-ended generative task without a single correct solution (as opposed to deletion tasks where the solution is clear). The “exact repair“ measure is a useful signal that shows that the model can recover the original ground truth sequence with some probability, but this metric is not something that can be saturated to 100% accuracy, since that would disregard alternative correct possibilities. While it is possible to create other metrics such as using LLM-as-a-judge to assess the quality of the corrected sentence, that introduces additional sources of noise (such as the model choice for the LLM-as-a-judge, and the threshold at which the output is scored as correct), hence we reported quantifiable measures such as the exact repair and changes in Levenshtein distance.

The sample size of the rollout evaluation should not be an issue. We will rerun the evaluation with 10,000 samples to get a better precision.

...(Continued in the next post, post 1/2)

...(Continued from the previous post, post 2/2)

Weakness 2 and question 2: isolating editing and cost. The point made in the paper is that the model attains new capabilities of editing its own context if required without sacrificing generative performance unlike diffusion LLMs. We do not claim that the improvements in performance are due to the model generating incorrect outputs before correcting them. In fact, it is desirable if the model does not generate an incorrect output so that edits are unnecessary. We expect an improved robustness from the models (as demonstrated in the edit diagnostics) from noise or mistakes in the context due to the edit capabilities. We also highlight the fact that this edit capability can be useful for downstream applications and opens opportunities for fine-tuning models on chain of thought traces that involve iterative modification of the context. We consider this out of scope and leave this for future work.

In terms of computational efficiency, we ran profiling and obtained the below:

ModelPerturb probPadded lenEffective raw lenPeak alloc (GiB)Peak reserved (GiB)Step time (s)
Qwen3 LoRAn/a2048204819.921.30.543
AutoIndexer LoRA1.030722302 (+12.4%)20.023.81.411
AutoIndexer LoRA0.03072204919.420.40.978

Here, gradient checkpointing was on for all models, and the batch size was 1. As shown, the forward pass time is longer by a factor of 2.6x for AutoIndexer, but with this additional time it parallelizes 50 edits, so the compute time per edit has improved 20x. Due to perturbation training, the sequence length is slightly longer than the default Qwen3 training (the frequency of edits depends on the user choice, but in our setting the sequence length increased by around 12%).

In terms of inference time, AutoIndexer’s inference time is slightly longer than Qwen3 (around +20%). However, this is without any inference time optimization in place, because the CuTeDSL acceleration was only applied to the attention kernel at training time, and our priorities were not on inference time acceleration.

ModelBatchPrefill (s)Decode ms/stepTokens/sPeak above weights (GiB)
Qwen310.04140.624.60.12
AutoIndexer10.48447.520.30.22
Qwen380.13340.9193.80.97
AutoIndexer80.58653.4144.11.75

On the MBPP score, the Qwen3 LoRA checkpoint yielded unstable results due to result formatting issues. We have rerun the LoRA training and evaluation, and obtained a more sensible number of 39.60, while also maintaining similar results for the other baselines. The claims in the paper still hold with the updated results.

ModelQwen3 8BQwen3 8B LoRA*LLaDA 8B InstructDream 7B InstructAutoIndexer 8B LoRA*
Decoding strategyARARDiffusionDiffusionCoE (Chain of Edits)
Editable contextNoNoLimitedLimitedYes
---
General tasks
ARC-Challenge (0)90.6190.4486.2688.2390.61
MMLU-Pro (0)42.3847.9032.6544.3947.90
---
Mathematics & science
GPQA-Diamond (5)35.3529.8020.7135.8630.81
GSM8K (5)86.9682.4175.7480.5980.67
MATH500 (4)52.2048.0017.0045.8049.00
---
Code
HumanEval (0)83.5471.9545.1252.4475.61
MBPP (3)67.4039.6037.6058.0064.00
Reviewer_VSrZOpenReview ↗

Response to rebuttal

Thanks for the response. The rerun MBPP number for the matched LoRA baseline (39.60) and the profiling results answer my questions about the matched comparison and the cost.

I'm fine with the framing that the paper claims editing ability without losing general performance, not that editing improves benchmark scores. Please make this explicit in the paper, so that readers don't attribute the HumanEval gain to editing.

My main reservation is unchanged. The self-correction evidence is still 100 synthetic corruptions per category, with no real model errors and no comparison against simply asking an AR model to revise its draft. The larger 10k-sample run and the edit benchmarks you mention would help here.

I'll keep my score at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.