- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose a new method for faster code generation: predicting start and end span regions for editing from attention and only editing this part. They train their models on Qwen-3-4B-Instruct-2507 and Qwen-3-30B-A3B-Instruct-2507 and get slightly higher scores on code editing benchmarks, such as CommitPackFT, CanItEdit and EditEval while using 0.71x as many autoregressively decoded tokens.
Critical strengths
The paper is easy to read and the presented graphs and images make sense. Their models slightly outperform baselines in accuracy on code editing benches. The training hyperparameters are well documented. The proposed algorithm of span copying action into an LLM is interesting, and the experiments show that the model can be trained to use this mechanism while having baseline code-generation performance after SFT and DPO.
Critical weaknesses
My main concern for this paper is that the evaluation does not show if the proposed learned pointer mechanism gives an advantage over much simpler alternatives. The authors correctly say that copied tokens still have to be prefetched into the KV cache, so reducing the number of autoregressively generated tokens does not translate into proportional wall-clock speedups. Despite autoregressive token ratios around 0.7x, the measured end-to-end serving gains are only around 1.1–1.3x on the reported benchmarks. This begs the question: is a much simpler alternative, such as emitting a diff instead of the complete file or ngram/prompt-lookup/other form of specdec be comparable in quality and speed, while being training-free approaches. None of these baselines are evaluated, so it is difficult to understand if the architectural and training novelty of the paper is justified.
Additionally, the evaluation is flawed. There was no comparison with a baseline, which only trains SFT/SFT+DPO without any copy head, thus, confounding the findings for increased code editing quality.
Also, in abstract, they report using 90.3% fewer autoregressive token generation and x7.13 faster generation time on a selected example, which is cherrypicking, since benchmark level comparison shows roughly 30-40% generated token reduction.
Questions for authors
How does PG-LLM compare in end-to-end latency, throughput and quality against generating a diff rather than the full edited file, prompt-lookup or n-gram speculative decoding, and stronger learned speculative decoding methods such as DFlash/DSPark?
Can the authors provide an ablation where the base Qwen model receives the same SFT and DPO data without the copy mechanism, to separate gains from the data from gains from the proposed architecture?
3 comments
Response to Reviewer nzaQ (2/2)
(continue Weakness 1 and Question 1 above)
In Table 1, the token ratios for DFlash and PG-LLM+DFlash measure the number of tokens directly generated by the target model relative to the ordinary LoRA baseline’s output length. PG-LLM’s specialized training may introduce a mismatch between the drafter and the target model’s prediction distributions, leading to more draft rejections and a slightly higher target-generated token ratio. Nevertheless, PG-LLM can directly copy selected source spans, bypassing neural drafting for those spans. This reduces the amount of content that DFlash’s shallow draft model needs to generate, enabling additional acceleration beyond standalone speculative decoding.
Overall, the experimental results show that PG-LLM, as a learned context-reuse method, can be combined with existing speculative decoding methods to achieve greater speedups.
Weakness 2 and Question 2: Separating training-data effects from the architecture.
We have added a generation-only LoRA SFT control using the same 22,614 training examples, with the full reference output as its target and no pointer head. Tables 1a–1b report all six benchmarks and also retain the original PG-LLM SFT results to distinguish the training stages.
The control experiments show that PG-LLM achieves code editing and code generation performance comparable to a conventional model fine-tuned on the same SFT data. Our main contribution is a learned pointer-generator mechanism that identifies and directly reuses relevant spans from the input context, accelerating decoding while preserving comparable overall output quality.
Table 1a. Comparison of code editing and generation performance across training settings with Qwen3-4B-Instruct-2507
| Model / training | CommitPackFT ↑ | CanItEdit ↑ | EditEval ↑ | LiveCodeBench ↑ | BigCodeBench-Hard ↑ | MBPP ↑ |
|---|---|---|---|---|---|---|
| Original model | 70.42 | 52.38 | 60.31 | 25.19 | 22.97 | 78.04 |
| Ordinary LoRA SFT | 72.30 | 55.24 | 60.82 | 23.66 | 22.30 | 77.25 |
| PG-LLM SFT | 71.83 | 54.29 | 57.22 | 21.37 | 18.24 | 74.60 |
| PG-LLM SFT+DPO | 71.36 | 56.19 | 62.89 | 24.43 | 23.65 | 79.10 |
Table 1b. Comparison of code editing and generation performance across training settings with Qwen3-30B-A3B-Instruct-2507
| Model / training | CommitPackFT ↑ | CanItEdit ↑ | EditEval ↑ | LiveCodeBench ↑ | BigCodeBench-Hard ↑ | MBPP ↑ |
|---|---|---|---|---|---|---|
| Original model | 82.16 | 64.76 | 73.20 | 43.51 | 41.22 | 88.89 |
| Ordinary LoRA SFT | 83.57 | 66.67 | 73.71 | 42.75 | 40.54 | 88.10 |
| PG-LLM SFT | 83.57 | 67.62 | 70.62 | 38.93 | 37.16 | 85.71 |
| PG-LLM SFT+DPO | 83.10 | 66.67 | 75.26 | 41.98 | 42.57 | 88.36 |
Thank you very much for your additional experiments.
With the SFT experiment, you show that performance of PG-LLM SFT is actually worse than regular SFT, which disables the claims that your approach increases quality of the model.
5-19% speed increase on top of DFlash/DSpark is actually a great result, but you did not provide comparisons with proposed simple baselines, such as ngram and prompt lookup, which would work wonders in code editing and, afaik, can also be used together with DFlash. Thus, I'm not convinced that additional complexity of training and serving PG-LLM are worth the 5% speed increases you claim, especially if there might be much simpler and efficient non-trainable approaches.
Thus, I won't consider raising my rating.
Response to Reviewer nzaQ (1/2)
Thank you for the direct assessment. We agree that stronger decoding baselines and a training-data control are necessary. We have added Prompt Lookup Decoding (PLD) and DFlash speculative-decoding baselines, a generation-only SFT control. Below, we clarify what these results establish address your concerns point by point.
Weakness 1 and Question 1: Comparison with alternative output and decoding methods.
We have added Prompt Lookup Decoding (PLD) and DFlash on the same two Qwen backbones, code-editing prompts, request sets, hardware, and batched workload. Tables 1a–1b report accuracy and end-to-end throughput gains relative to ordinary LoRA autoregressive decoding, including lookup, drafting, verification, pointer operations, and copied-span prefill.
We agree that PG-LLM alone is slower on these code-editing workloads. An important design choice is our conservative, AST-aligned copy supervision: it favors complete structural units over arbitrary unchanged fragments. The original granularity ablation supports this tradeoff: block-level traces improve editing accuracy from 58.08% to 61.11%, while increasing the autoregressive token ratio from 0.57 to 0.68. Therefore, PG-LLM’s lower speedup mainly stems from the conservative copy coverage used during pointer-head training, which favors complete, semantically coherent AST-aligned code spans. At the same token ratio, PG-LLM remains competitive in terms of speedup.
Crucially, learned span reuse can complement speculative decoding. We combine PG-LLM with DFlash by drafting ordinary text and executing pointer-selected copies at copy boundaries. The combination retains PG-LLM's benchmark accuracy and reaches 2.16–2.64× speedup, an additional 5.88%–19.71% over standalone DFlash.
Table 1a. Comparison across decoding methods on code editing benchmarks with Qwen3-4B-Instruct-2507
& \text{CommitPackFT} & \text{CanItEdit} & \text{EditEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Ordinary LoRA SFT} \\\\ \text{PG-LLM} \\\\ \text{PLD} \\\\ \text{DFlash} \\\\ \text{PG-LLM+DFlash} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 72.30 & 100.00 & 1.00\times \\\\ 71.36 & 62.62 & 1.21\times \\\\ 72.30 & 29.64 & 1.76\times \\\\ 72.30 & 15.62 & 2.24\times \\\\ 71.36 & 17.45 & \mathbf{2.64\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 55.24 & 100.00 & 1.00\times \\\\ 56.19 & 64.88 & 1.28\times \\\\ 55.24 & 21.83 & 1.79\times \\\\ 55.24 & 14.68 & 2.18\times \\\\ 56.19 & 16.24 & \mathbf{2.57\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 60.82 & 100.00 & 1.00\times \\\\ 62.89 & 83.21 & 1.12\times \\\\ 60.82 & 41.72 & 1.67\times \\\\ 60.82 & 16.13 & 2.12\times \\\\ 62.89 & 18.50 & \mathbf{2.31\times} \end{array} \end{array}$$ **Table 1b. Comparison across decoding methods on code editing benchmarks with Qwen3-30B-A3B-Instruct-2507** $$\begin{array}{l|c|c|c} & \text{CommitPackFT} & \text{CanItEdit} & \text{EditEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Ordinary LoRA SFT} \\\\ \text{PG-LLM} \\\\ \text{PLD} \\\\ \text{DFlash} \\\\ \text{PG-LLM+DFlash} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 83.57 & 100.00 & 1.00\times \\\\ 83.10 & 64.83 & 1.18\times \\\\ 83.57 & 29.71 & 1.74\times \\\\ 83.57 & 14.71 & 2.11\times \\\\ 83.10 & 16.25 & \mathbf{2.48\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 66.67 & 100.00 & 1.00\times \\\\ 66.67 & 66.38 & 1.26\times \\\\ 66.67 & 23.68 & 1.83\times \\\\ 66.67 & 14.29 & 2.08\times \\\\ 66.67 & 17.23 & \mathbf{2.49\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 73.71 & 100.00 & 1.00\times \\\\ 75.26 & 82.42 & 1.13\times \\\\ 73.71 & 41.57 & 1.62\times \\\\ 73.71 & 15.38 & 2.04\times \\\\ 75.26 & 19.74 & \mathbf{2.16\times} \end{array} \end{array}$$ (continue below)