- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors argue that autoregressive language models can not directly modify previously generated tokens during decoding. To address this, they propose AutoIndexer, a flexible-order decoding method based on Chains of Edits. The method uses marker and cursor prediction heads to decide when and where to insert, delete, or replace tokens. It also introduces edit-aware positional encoding to support these operations while preserving KV caching. During training, the model learns to reverse randomly generated edit operations applied to clean text. The results show that AutoIndexer largely preserves the original performance and achieves competitive results against diffusion models. Additional experiments using synthetic text corruptions show that the model can detect and repair some errors through autonomous editing.
Critical strengths
The proposed decoding is interesting. It allows autoregressive models to insert, delete, and replace previously generated tokens. The edit-aware positional encoding might be a useful technical contribution that supports KV-cache reuse. The method can thus be adapted from pretrained models using LoRA.
The experimental results are generally positive. AutoIndexer largely preserves the performance of the original model on standard benchmarks. It performs competitively against diffusion models. The free-generation experiments also show that the model can independently initiate and perform edits.
Critical weaknesses
The novelty needs clearer positioning against prior text-editing work (Reviser by Diab, 2026). The proposed perturbation training and the learning to edit text through actions and constructed edit trajectories have been explored before [Gu et al., 2019; Shi et al., 2022]. The fused kernel and Chain-of-Edits RoPE are interesting ideas, but their individual benefits are not demonstrated.
The experiments do not sufficiently demonstrate the practical benefits of self-correction. The general benchmark results mainly show that AutoIndexer preserves the original performance, but not an improvement. The editing experiments use synthetic errors and lack direct comparisons with existing editing or self-refinement baselines. The autonomous editing results are also limited, especially for insertion and substitution. It remains unclear whether the method can effectively correct mistakes in its own generated outputs or improve performance on real tasks as stated in the introduction where autoregressive LLMs often fail.
The experimental scope and analysis are limited. The experiments mainly focus on Qwen3-8B, and the autonomous editing evaluation uses only 100 examples per condition. The paper also lacks ablation studies to isolate the contributions of its main components. It also does not report training cost or inference speed. Since the method adds editing actions, prediction heads, and specialized attention, these measurements are helpful.
Questions for authors
- It is not accurate to state that autoregressive LLMs are not able to refine their outputs. They can reconsider and refine answers internally (e.g., in thoughts) before producing the final response to the user. In this sense, what practical advantage does AutoIndexer offer over these approaches?
- How does its editing performance compare with existing editing or self-refinement methods on established benchmarks?
- What is the individual contribution of Chain-of-Edits RoPE?
- Table 5 shows limited success for insertion and substitution. What are the main reasons for these failures?
3 comments
...(Continued from the previous post, post 2/2)
Weakness 3. The sample size of the rollout evaluation should not be an issue. We will rerun the evaluation with 10,000 samples to get a better precision. In terms of ablation studies, the main components (perturbation training, marker prediction head, cursor prediction head and Chain-of-edits RoPE) are only effective when combined together. We confirmed that without perturbation training and marker/cursor predictions turned off, the Chain-of-edits RoPE attention kernel behaves equivalently to a standard attention kernel.
In terms of computational efficiency, we ran profiling and obtained the below:
| Model | Perturb prob | Padded len | Effective raw len | Peak alloc (GiB) | Peak reserved (GiB) | Step time (s) |
|---|---|---|---|---|---|---|
| Qwen3 LoRA | n/a | 2048 | 2048 | 19.9 | 21.3 | 0.543 |
| AutoIndexer LoRA | 1.0 | 3072 | 2302 (+12.4%) | 20.0 | 23.8 | 1.411 |
| AutoIndexer LoRA | 0.0 | 3072 | 2049 | 19.4 | 20.4 | 0.978 |
Here, gradient checkpointing was on for all models, and the batch size was 1. As shown, the forward pass time is longer by a factor of 2.6x for AutoIndexer, but with this additional time it parallelizes 50 edits, so the compute time per edit has improved 20x. Due to perturbation training, the sequence length is slightly longer than the default Qwen3 training (the frequency of edits depends on the user choice, but in our setting the sequence length increased by around 12%).
In terms of inference time, AutoIndexer’s inference time is slightly longer than Qwen3 (around +20%). However, this is without any inference time optimization in place, because the CuTeDSL acceleration was only applied to the attention kernel at training time, and our priorities were not on inference time acceleration.
| Model | Batch | Prefill (s) | Decode ms/step | Tokens/s | Peak above weights (GiB) |
|---|---|---|---|---|---|
| Qwen3 | 1 | 0.041 | 40.6 | 24.6 | 0.12 |
| AutoIndexer | 1 | 0.484 | 47.5 | 20.3 | 0.22 |
| Qwen3 | 8 | 0.133 | 40.9 | 193.8 | 0.97 |
| AutoIndexer | 8 | 0.586 | 53.4 | 144.1 | 1.75 |
Question 1. We agree that reasoning models can refine an answer internally. That reasoning, however, refines a draft that is not part of the output, and the final answer must then be written out again. Tokens already emitted into the output cannot be changed in place, resulting in more token usage, which is the limitation we address. The advantage of our method is that minor corrections can be made without regenerating the entire text.
Question 3. Chain-of-Edits RoPE is what lets the model reassign token positions after each edit so that tokens are positioned in display order at each step, allowing parallel training over N number of context edits in one forward pass, and without having to recompute the keys and values at inference time. Without this mechanism, the model would have to reorder the tokens after each edit operation and perform N numbers of forward passes at training time, and have to recompute the keys and values over the entire sequence so far N number of times at inference time.
I appreciate the detailed response. The technical novelty is now much clearer, particularly around the position updates and KV-cache reuse. Action-based text editing is not really a novel concept, but the decoding architecture of AutoIndexer seems to be different from what has gone before. I also agree that an exact match is going to be a rather poor proxy for the quality of the insertion and substitution.
However my main issues remain. There are still no direct comparisons with existing editing methods. Although the paper evaluates standard benchmarks, it does not show whether editing improves performance on real tasks, such as correcting errors in generated code. Note that the introduction motivates the work with applications like code correction. My question was not on the sota performance, but on whether it actually offers meaningful advantages over existing approaches. The new efficiency results are useful but still do not establish that it is better. The larger evaluation that is planned may help, yet all I can go on at the moment is based on what I can see.
Overall, it does clarify a few things, but some issues are still there. So I intend to keep my current rating.
We thank the reviewer for the feedback and for noting the interest of the decoding scheme, the positional encoding, and the KV-cache reuse.
TL;DR: (1) Novelty: Unlike prior editing methods, AutoIndexer supports learned cursor prediction, display-position updates, and exact KV-cache reuse across edits. (2) Practical benefit: Unlike internal reasoning, AutoIndexer directly revises previously generated output without regenerating the full text; comparisons with existing editing methods are future work. (3) Chain-of-Edits RoPE: Enables multiple edits in one training forward pass and avoids KV recomputation at inference. (4) Editing results: Exact-match metrics underestimate open-ended insertion/substitution repairs; we plan to expand evaluation to 10K samples. (5) Ablations and efficiency: training takes 2.6x longer per step, while inference has ~20% overhead without optimization.
Weakness 1: novelty and positioning. The paper positions the method against this work in Section 4 and Table 1. The closest prior work differs on the points that define our contribution. Gu et al. (2019) and the other insertion/deletion decoders are non-autoregressive, so token positions shift between steps and the KV cache cannot be reused. Reviser (Diab, 2026) predicts cursor locations in absolute token shifts (combining tokens that indicate shifts by 1, 2, 4, 8, 16, 32, etc.), and has to refresh the keys and values after each edit, while AutoIndexer localizes edits with a pointer network, reassigns positional encodings after each edit, and keeps an exact KV cache across multiple edits. As for the Shi et al., 2022 paper, it is ambiguous which paper the reviewer is referring to. Assuming it is the “Text Editing as Imitation Game“ paper, they also re-encode the edited text after each edit, whereas for AutoIndexer, edits happen inside one decoding stream with exact KV-cache reuse and updated positional encoding. Neither prior paper decouples generation order from display order of tokens. We will revise the related work section and elaborate on these distinctions.
On the components, the fused kernel is a training-efficiency device. Chain-of-Edits RoPE is not an optional add-on and is a core part of the method, since the standard RoPE factorization only works for static sequences (Section 2.3) and cannot express multi-edit position shifts in one packed forward pass.
Weakness 2 and question 2: evaluation of edit capabilities. Our model fine-tuning recipe for edits employs a generic diffusion-like denoising loop rather than fine-tuning directly on edit datasets. The point made in the paper is that the model attains new capabilities of editing its own context if required without sacrificing generative performance, unlike diffusion LLMs. To this end, we position our work as an alternative paradigm to diffusion training for LLMs, as opposed to as a frontier editing model. Having said this, additional experiments on edit benchmarks would strengthen our work. We consider your suggestion to be useful feedback and aim to include these comparisons for a camera-ready paper.
Weakness 2 and question 4: analysis of insertion and substitution cases. Successes for insertion and substitution are naturally more difficult to measure, since it is an open-ended generative task without a single correct solution (as opposed to deletion tasks where the solution is clear). The “exact repair“ measure is a useful signal that shows that the model can recover the original ground truth sequence with some probability, but this metric is not something that can be saturated to 100% accuracy, since that would disregard alternative correct possibilities. While it is possible to create other metrics such as using LLM-as-a-judge to assess the quality of the corrected sentence, that introduces additional sources of noise (such as the model choice for the LLM-as-a-judge, and the threshold at which the output is scored as correct), hence we reported quantifiable measures such as the exact repair and changes in Levenshtein distance.
...(Continued in the next post, post 1/2)