acceptodds
Under review as a conference paper at ICLR 2027

Do Not Regenerate What is in Your Context: Instant Context Reuse via a Pointer Head

Abstract

Large language models increasingly process long-context inputs containing content that may reappear in their outputs, making redundant token-by-token generation a source of decoding overhead. Existing efficient decoding methods, including multi-token prediction and speculative decoding, reduce sequential computation through parallel prediction or draft verification but do not explicitly learn to locate and reuse such content. We propose the Pointer-Generator Large Language Model (PG-LLM), which turns context reuse into a model-native generation action. PG-LLM emits a learned control token when reuse is appropriate and uses an attention-based pointer mechanism to predict the start and end positions of a relevant source span. The selected span is inserted into the output before autoregressive generation resumes. We train PG-LLM at 4B and 30B scales. Across three code editing benchmarks, PG-LLM 30B uses as many autoregressively decoded tokens as its base model on average and improves editing accuracy by 1.64 percentage points, while preserving comparable overall performance across three code generation benchmarks. With a vLLM serving implementation, PG-LLM improves request throughput over the corresponding base model by – at 4B and – at 30B. These results demonstrate that model-native context localization and reuse can reduce autoregressive decoding and deliver practical serving speedups while maintaining comparable overall code generation performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Efficient Decoding, Context Reuse, Pointer Networks, Code Editing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nzaQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a new method for faster code generation: predicting start and end span regions for editing from attention and only editing this part. They train their models on Qwen-3-4B-Instruct-2507 and Qwen-3-30B-A3B-Instruct-2507 and get slightly higher scores on code editing benchmarks, such as CommitPackFT, CanItEdit and EditEval while using 0.71x as many autoregressively decoded tokens.

Critical strengths

The paper is easy to read and the presented graphs and images make sense. Their models slightly outperform baselines in accuracy on code editing benches. The training hyperparameters are well documented. The proposed algorithm of span copying action into an LLM is interesting, and the experiments show that the model can be trained to use this mechanism while having baseline code-generation performance after SFT and DPO.

Critical weaknesses

My main concern for this paper is that the evaluation does not show if the proposed learned pointer mechanism gives an advantage over much simpler alternatives. The authors correctly say that copied tokens still have to be prefetched into the KV cache, so reducing the number of autoregressively generated tokens does not translate into proportional wall-clock speedups. Despite autoregressive token ratios around 0.7x, the measured end-to-end serving gains are only around 1.1–1.3x on the reported benchmarks. This begs the question: is a much simpler alternative, such as emitting a diff instead of the complete file or ngram/prompt-lookup/other form of specdec be comparable in quality and speed, while being training-free approaches. None of these baselines are evaluated, so it is difficult to understand if the architectural and training novelty of the paper is justified.

Additionally, the evaluation is flawed. There was no comparison with a baseline, which only trains SFT/SFT+DPO without any copy head, thus, confounding the findings for increased code editing quality.

Also, in abstract, they report using 90.3% fewer autoregressive token generation and x7.13 faster generation time on a selected example, which is cherrypicking, since benchmark level comparison shows roughly 30-40% generated token reduction.

Questions for authors

How does PG-LLM compare in end-to-end latency, throughput and quality against generating a diff rather than the full edited file, prompt-lookup or n-gram speculative decoding, and stronger learned speculative decoding methods such as DFlash/DSPark?

Can the authors provide an ablation where the base Qwen model receives the same SFT and DPO data without the copy mechanism, to separate gains from the data from gains from the proposed architecture?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nzaQ (1/2)

Thank you for the direct assessment. We agree that stronger decoding baselines and a training-data control are necessary. We have added Prompt Lookup Decoding (PLD) and DFlash speculative-decoding baselines, a generation-only SFT control. Below, we clarify what these results establish address your concerns point by point.


Weakness 1 and Question 1: Comparison with alternative output and decoding methods.

We have added Prompt Lookup Decoding (PLD) and DFlash on the same two Qwen backbones, code-editing prompts, request sets, hardware, and batched workload. Tables 1a–1b report accuracy and end-to-end throughput gains relative to ordinary LoRA autoregressive decoding, including lookup, drafting, verification, pointer operations, and copied-span prefill.

We agree that PG-LLM alone is slower on these code-editing workloads. An important design choice is our conservative, AST-aligned copy supervision: it favors complete structural units over arbitrary unchanged fragments. The original granularity ablation supports this tradeoff: block-level traces improve editing accuracy from 58.08% to 61.11%, while increasing the autoregressive token ratio from 0.57 to 0.68. Therefore, PG-LLM’s lower speedup mainly stems from the conservative copy coverage used during pointer-head training, which favors complete, semantically coherent AST-aligned code spans. At the same token ratio, PG-LLM remains competitive in terms of speedup.

Crucially, learned span reuse can complement speculative decoding. We combine PG-LLM with DFlash by drafting ordinary text and executing pointer-selected copies at copy boundaries. The combination retains PG-LLM's benchmark accuracy and reaches 2.16–2.64× speedup, an additional 5.88%–19.71% over standalone DFlash.

Table 1a. Comparison across decoding methods on code editing benchmarks with Qwen3-4B-Instruct-2507

& \text{CommitPackFT} & \text{CanItEdit} & \text{EditEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Ordinary LoRA SFT} \\\\ \text{PG-LLM} \\\\ \text{PLD} \\\\ \text{DFlash} \\\\ \text{PG-LLM+DFlash} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 72.30 & 100.00 & 1.00\times \\\\ 71.36 & 62.62 & 1.21\times \\\\ 72.30 & 29.64 & 1.76\times \\\\ 72.30 & 15.62 & 2.24\times \\\\ 71.36 & 17.45 & \mathbf{2.64\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 55.24 & 100.00 & 1.00\times \\\\ 56.19 & 64.88 & 1.28\times \\\\ 55.24 & 21.83 & 1.79\times \\\\ 55.24 & 14.68 & 2.18\times \\\\ 56.19 & 16.24 & \mathbf{2.57\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 60.82 & 100.00 & 1.00\times \\\\ 62.89 & 83.21 & 1.12\times \\\\ 60.82 & 41.72 & 1.67\times \\\\ 60.82 & 16.13 & 2.12\times \\\\ 62.89 & 18.50 & \mathbf{2.31\times} \end{array} \end{array}$$ **Table 1b. Comparison across decoding methods on code editing benchmarks with Qwen3-30B-A3B-Instruct-2507** $$\begin{array}{l|c|c|c} & \text{CommitPackFT} & \text{CanItEdit} & \text{EditEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Ordinary LoRA SFT} \\\\ \text{PG-LLM} \\\\ \text{PLD} \\\\ \text{DFlash} \\\\ \text{PG-LLM+DFlash} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 83.57 & 100.00 & 1.00\times \\\\ 83.10 & 64.83 & 1.18\times \\\\ 83.57 & 29.71 & 1.74\times \\\\ 83.57 & 14.71 & 2.11\times \\\\ 83.10 & 16.25 & \mathbf{2.48\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 66.67 & 100.00 & 1.00\times \\\\ 66.67 & 66.38 & 1.26\times \\\\ 66.67 & 23.68 & 1.83\times \\\\ 66.67 & 14.29 & 2.08\times \\\\ 66.67 & 17.23 & \mathbf{2.49\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 73.71 & 100.00 & 1.00\times \\\\ 75.26 & 82.42 & 1.13\times \\\\ 73.71 & 41.57 & 1.62\times \\\\ 73.71 & 15.38 & 2.04\times \\\\ 75.26 & 19.74 & \mathbf{2.16\times} \end{array} \end{array}$$ (continue below)
AuthorsOpenReview ↗

Response to Reviewer nzaQ (2/2)

(continue Weakness 1 and Question 1 above)

In Table 1, the token ratios for DFlash and PG-LLM+DFlash measure the number of tokens directly generated by the target model relative to the ordinary LoRA baseline’s output length. PG-LLM’s specialized training may introduce a mismatch between the drafter and the target model’s prediction distributions, leading to more draft rejections and a slightly higher target-generated token ratio. Nevertheless, PG-LLM can directly copy selected source spans, bypassing neural drafting for those spans. This reduces the amount of content that DFlash’s shallow draft model needs to generate, enabling additional acceleration beyond standalone speculative decoding.

Overall, the experimental results show that PG-LLM, as a learned context-reuse method, can be combined with existing speculative decoding methods to achieve greater speedups.


Weakness 2 and Question 2: Separating training-data effects from the architecture.

We have added a generation-only LoRA SFT control using the same 22,614 training examples, with the full reference output as its target and no pointer head. Tables 1a–1b report all six benchmarks and also retain the original PG-LLM SFT results to distinguish the training stages.

The control experiments show that PG-LLM achieves code editing and code generation performance comparable to a conventional model fine-tuned on the same SFT data. Our main contribution is a learned pointer-generator mechanism that identifies and directly reuses relevant spans from the input context, accelerating decoding while preserving comparable overall output quality.

Table 1a. Comparison of code editing and generation performance across training settings with Qwen3-4B-Instruct-2507

Model / trainingCommitPackFT ↑CanItEdit ↑EditEval ↑LiveCodeBench ↑BigCodeBench-Hard ↑MBPP ↑
Original model70.4252.3860.3125.1922.9778.04
Ordinary LoRA SFT72.3055.2460.8223.6622.3077.25
PG-LLM SFT71.8354.2957.2221.3718.2474.60
PG-LLM SFT+DPO71.3656.1962.8924.4323.6579.10

Table 1b. Comparison of code editing and generation performance across training settings with Qwen3-30B-A3B-Instruct-2507

Model / trainingCommitPackFT ↑CanItEdit ↑EditEval ↑LiveCodeBench ↑BigCodeBench-Hard ↑MBPP ↑
Original model82.1664.7673.2043.5141.2288.89
Ordinary LoRA SFT83.5766.6773.7142.7540.5488.10
PG-LLM SFT83.5767.6270.6238.9337.1685.71
PG-LLM SFT+DPO83.1066.6775.2641.9842.5788.36

Thank you very much for your additional experiments.

With the SFT experiment, you show that performance of PG-LLM SFT is actually worse than regular SFT, which disables the claims that your approach increases quality of the model.

5-19% speed increase on top of DFlash/DSpark is actually a great result, but you did not provide comparisons with proposed simple baselines, such as ngram and prompt lookup, which would work wonders in code editing and, afaik, can also be used together with DFlash. Thus, I'm not convinced that additional complexity of training and serving PG-LLM are worth the 5% speed increases you claim, especially if there might be much simpler and efficient non-trainable approaches.

Thus, I won't consider raising my rating.

Reviewer_ANEvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PG-LLM, a model that reduces autoregressive decoding by reusing content already available in the input context. The model uses a learned control token and an attention-based pointer mechanism to predict the start and end positions of a reusable source span. The authors train 4B and 30B models using AST-aligned copy traces and preference optimization. Experiments on code editing benchmarks show that PG-LLM reduces the number of autoregressively decoded tokens while maintaining or improving editing accuracy. The vLLM implementation also improves request throughput by 1.12–1.28×.

Critical strengths
  1. The paper proposes a method that allows LLMs to directly reuse existing content in the input context instead of generating the same tokens again. The pointer mechanism is integrated into the model, and the AST-aligned training strategy helps the model learn when and what to copy.
  2. The paper provides experimental results across different model sizes and benchmarks. In particular, PG-LLM 30B reduces autoregressive token generation by approximately 29% while improving code editing accuracy by 1.64 percentage points.
Critical weaknesses
  1. The experiments mainly compare PG-LLM with the original base models. However, PG-LLM also receives additional SFT and DPO training. Therefore, it is unclear whether the improvement in code editing accuracy comes from the pointer mechanism or the additional training.
  2. The paper uses long-context scenarios as a main motivation, but the evaluation focuses on code-related tasks. In particular, the average source code length in the training data is relatively short (approximately 204 tokens), and the paper does not provide an analysis of performance across different context lengths.
Questions for authors
  1. Could the authors provide code editing and code generation results from a model trained on the same SFT and DPO data but without the pointer head?
  2. How does the performance of PG-LLM change as the input context length increases?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ANEv (1/2)

Thank you for identifying the need for a training-data control and a longer-context evaluation. We have added both experiments and clarify their implications below.


Weakness 1 and Question 1: Same-data training without the pointer head.

We have added a generation-only LoRA SFT control using the same 22,614 training examples, with the full reference output as its target and no pointer head. Tables 1a–1b report all six benchmarks and also retain the original PG-LLM SFT results to distinguish the training stages.

The control experiments show that PG-LLM achieves code editing and code generation performance comparable to a conventional model fine-tuned on the same SFT data. Our main contribution is a learned pointer-generator mechanism that identifies and directly reuses relevant spans from the input context, accelerating decoding while preserving comparable overall output quality.

Table 1a. Comparison of code editing and generation performance across training settings with Qwen3-4B-Instruct-2507

Model / trainingCommitPackFT ↑CanItEdit ↑EditEval ↑LiveCodeBench ↑BigCodeBench-Hard ↑MBPP ↑
Original model70.4252.3860.3125.1922.9778.04
Ordinary LoRA SFT72.3055.2460.8223.6622.3077.25
PG-LLM SFT71.8354.2957.2221.3718.2474.60
PG-LLM SFT+DPO71.3656.1962.8924.4323.6579.10

Table 1b. Comparison of code editing and generation performance across training settings with Qwen3-30B-A3B-Instruct-2507

Model / trainingCommitPackFT ↑CanItEdit ↑EditEval ↑LiveCodeBench ↑BigCodeBench-Hard ↑MBPP ↑
Original model82.1664.7673.2043.5141.2288.89
Ordinary LoRA SFT83.5766.6773.7142.7540.5488.10
PG-LLM SFT83.5767.6270.6238.9337.1685.71
PG-LLM SFT+DPO83.1066.6775.2641.9842.5788.36

Weakness 2 and Question 2: Longer contexts and variable-length reuse.

Summary. PG-LLM outperforms Prompt Lookup Decoding (PLD) and DFlash in speed via variable-length span reuse on GovReport, with a remaining accuracy tradeoff.

We have added a long-context evidence-extraction task derived from GovReport. Preserving the original report-level splits, we construct 3,000 training examples and 261 test examples. Each input contains an 8K–16K-token report and a review request; the reference output concatenates four requested source sections verbatim in the requested order. Two short spans of 200–600 tokens alternate with two long spans of 1,200–3,500 tokens, producing 3K–6.5K-token outputs. Ordinary LoRA and PG-LLM receive the same task SFT data, and accuracy requires an exact match of the complete output.

Table 1a. Comparison on GovReport mixed-length evidence extraction with Qwen3-4B-Instruct-2507

MethodAcc ↑Token ↓Speedup ↑
Ordinary LoRA SFT65.13100.001.00×
PG-LLM60.151.137.24×
PLD (K=1024)65.132.175.87×
DFlash65.138.362.56×

Table 1b. Comparison on GovReport mixed-length evidence extraction with Qwen3-30B-A3B-Instruct-2507

MethodAcc ↑Token ↓Speedup ↑
Ordinary LoRA SFT75.48100.001.00×
PG-LLM71.261.067.63×
PLD (K=1024)75.481.866.86×
DFlash75.487.812.74×

As shown in Tables 1a–1b, PG-LLM achieves 7.24×/7.63× end-to-end speedup at 4B/30B, compared with 5.87×/6.86× for Prompt Lookup Decoding (PLD) and 2.56×/2.74× for DFlash. Here, PLD uses a maximum draft length of K=1024, where K limits the number of lookup-proposed tokens submitted for verification in each round. DFlash still uses a neural drafter to produce candidate text, while PLD and PG-LLM obtain reusable content directly from the context. This additional drafting work provides a plausible explanation for DFlash’s lower speedup on this reuse-heavy workload.

A key advantage of PG-LLM is its ability to select task-dependent span boundaries and thereby adapt the reuse length. When short spans contain hundreds of tokens and long spans contain thousands, the fixed-K PLD configurations evaluated here face a tradeoff between verification calls and candidate utilization.

...(continue below)

AuthorsOpenReview ↗

Response to Reviewer ANEv (2/2)

(continue Weakness 2 and Question 2 above)

Tables 2a–2b illustrate this tradeoff. K=64 maintains high block utilization but requires 19.62×/19.36× as many copy/verification block forwards as PG-LLM. K=2048 reduces the number of calls, but normalized block utilization falls to 48.29%/49.31%, with PG-LLM normalized to 100%, as more verified candidates are discarded. K=1024 is the fastest of the three evaluated configurations, but remains slower than PG-LLM. By learning to identify relevant source spans and predict their start and end positions, PG-LLM accommodates both short and long reuse segments and achieves the highest speedup among the evaluated methods.

Table 2a. Block processing and speed of PG-LLM and PLD with different window sizes on GovReport with Qwen3-4B-Instruct-2507

MethodBlock forwards (PG=100%) ↓Block utilization (PG=100%) ↑Relative speed (PG=100%) ↑
PG-LLM100.00%100.00%100.00%
PLD, K=641,962.36%96.66%70.72%
PLD, K=1024179.37%67.07%81.08%
PLD, K=2048124.62%48.29%64.64%

Table 2b. Block processing and speed of PG-LLM and PLD with different window sizes on GovReport with Qwen3-30B-A3B-Instruct-2507

MethodBlock forwards (PG=100%) ↓Block utilization (PG=100%) ↑Relative speed (PG=100%) ↑
PG-LLM100.00%100.00%100.00%
PLD, K=641,936.49%97.97%79.69%
PLD, K=1024176.09%68.31%89.91%
PLD, K=2048122.03%49.31%71.69%

We also acknowledge that PG-LLM currently has limitations in output quality, with slightly lower accuracy than the other methods. Limited training data and diversity may contribute to this gap. More diverse requests and carefully designed span-boundary supervision may improve correctness while retaining the efficiency benefits of adaptive span reuse.

Overall, the long-context results demonstrate the effectiveness of PG-LLM’s ability to identify reusable spans semantically and adaptively determine their lengths, achieving higher speedups, although further training is needed to improve accuracy.

Reviewer_mQ6QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PG-LLM, which augments a pretrained decoder with a copy control token and two pointer predictions. The first selects a source start; the selected source representation is injected into the control-token embedding to condition the endpoint prediction. The selected span is inserted and prefilled before generation resumes. Training combines Python AST-aligned copy trajectories, LoRA adaptation, and DPO. Across three code-editing datasets, the 30B DPO model reports a 0.71 autoregressive token ratio and a 1.64 percentage-point increase in average accuracy relative to its unmodified base model. A vLLM implementation reports throughput gains of 1.12–1.28 times at 4B and 1.13–1.26 times at 30B. Separate generation evaluations show that DPO largely repairs the degradation introduced by SFT.

Critical strengths
  1. The system implements a concrete and useful operation. Start-conditioned endpoint selection, source-only copying, and copied-span prefill are described clearly in Sections 3 and A. The design preserves the full revised-code interface and makes reuse a learned decision. The integration with batched vLLM serving is more convincing than token-count savings alone.

  2. The evaluation identifies real tradeoffs. Table 2 shows substantial generation degradation after SFT and its recovery after DPO. Table 4 shows that maximizing copying through line-level traces can hurt correctness. The paper also recognizes that copying mistakes do not explain all generation errors: MBPP performance drops even without copy triggers.

  3. Several common efficiency ambiguities are handled carefully. Equations 9 and 18 define corpus-level autoregressive counts, include copy control tokens, and distinguish the benchmark ratio from the qualitative example's source-normalized ratio. Sections 3.3 and E explicitly include span prefill and pointer costs in the discussion, acknowledge different output lengths, and distinguish batched throughput from the unusually favorable single example.

Critical weaknesses
  1. The baseline set does not establish the value of learned span reuse relative to existing context-based acceleration. Tables 1–3 compare PG-LLM with an unmodified autoregressive base and a slow Transformers reference implementation. They do not compare with a context lookup method under the same engine and workload. Prompt Lookup Decoding and LLMA: Inference with Reference already obtain candidate spans from available text and verify them in parallel. These methods do not learn the same operation, but they address the same practical source of redundant decoding. The claim in Section 2.1 that existing approaches do not explicitly exploit available spans needs this distinction. Without a matched comparison, the reported 12–28% throughput improvement establishes a benefit over ordinary decoding, but leaves the practical reason to retrain a model and maintain custom engine hooks unclear. The paper does cite earlier learned span-copying work; its contribution should be positioned around the particular LLM training and serving implementation rather than the general span-reuse idea.

  2. The reported accuracy benefit is not isolated from post-training, and the modest gains lack paired uncertainty. The authors correctly acknowledge this limitation in Sections 5.1 and 6. Nevertheless, the headline combines architecture and training gains against an untouched base. A generation-only model adapted with comparable editing/generation SFT and preference data is the relevant control for attributing quality gains. For 30B, the rounded scores in Table 1 correspond to net gains of approximately 2, 2, and 4 correct examples on datasets of 213, 105, and 194 items. These are useful observations, but not strong evidence of a general quality improvement without per-example paired results. One third of the average also uses the same judge family employed in editing preference construction; no judge prompt, agreement assessment, or alternative adjudication is reported in the supplied PDF. The two functional-test datasets are a useful counterweight, so this is a concern about precision and attribution, not evidence that the gains are invalid.

  3. The serving accuracy–throughput tradeoff is unresolved for the guarded workload. Appendix E.2 stops an EditEval pointer request after repeated selection of the same span, while throughput counts completed requests per wall time. This is a reasonable safeguard, and the paper discloses it. However, the guard's activation rate and the functional correctness of the actual timed vLLM outputs are not reported. Appendix E.4 also allows small decoding differences across implementations. It is therefore unclear whether Table 3 measures the same quality as Table 1 or includes faster termination of failed outputs. Reporting guard activations and correctness for those timed outputs would resolve this directly. Error bars across the three throughput runs would also help assess the smallest gains.

Questions for authors
  1. Do you have existing context-lookup/speculative-decoding results with the same Qwen models, vLLM configuration, prompts, and request batches? If not, please explain which claimed benefit is intended to go beyond this class of methods and narrow the efficiency comparison accordingly.
  2. Was a generation-only SFT/DPO control trained on comparable data? From the existing benchmark outputs, can you report paired correctness changes or confidence intervals and release the CommitPackFT judging prompt and adjudication settings?
  3. For each timed EditEval run, how many requests triggered the repeated-span guard, and what was functional accuracy? Were the Table 1 outputs produced with the same guard and engine? Please also provide the variation across the three timing runs.
  4. What are the pointer dimension, trace-filter thresholds, and benchmark output budgets? Are the detailed configurations and held-out example identifiers available in the submitted material?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mQ6Q (1/2)

Thank you for the careful review and concrete suggestions. We have added context-lookup and learned speculative-decoding baselines, a generation-only SFT control. Below, we clarify what these results establish address your concerns point by point.


Weakness 1 and question 1: Comparison with context lookup and speculative decoding.

We have added Prompt Lookup Decoding (PLD) and DFlash on the same two Qwen backbones, code-editing prompts, request sets, hardware, and batched workload. Tables 1a–1b report accuracy and end-to-end throughput gains relative to ordinary LoRA autoregressive decoding, including lookup, drafting, verification, pointer operations, and copied-span prefill.

We agree that PG-LLM alone is slower on these code-editing workloads. An important design choice is our conservative, AST-aligned copy supervision: it favors complete structural units over arbitrary unchanged fragments. The original granularity ablation supports this tradeoff: block-level traces improve editing accuracy from 58.08% to 61.11%, while increasing the autoregressive token ratio from 0.57 to 0.68. Therefore, PG-LLM’s lower speedup mainly stems from the conservative copy coverage used during pointer-head training, which favors complete, semantically coherent AST-aligned code spans. At the same token ratio, PG-LLM remains competitive in terms of speedup.

Crucially, learned span reuse can complement speculative decoding. We combine PG-LLM with DFlash by drafting ordinary text and executing pointer-selected copies at copy boundaries. The combination retains PG-LLM's benchmark accuracy and reaches 2.16–2.64× speedup, an additional 5.88%–19.71% over standalone DFlash.

Table 1a. Comparison across decoding methods on code editing benchmarks with Qwen3-4B-Instruct-2507

& \text{CommitPackFT} & \text{CanItEdit} & \text{EditEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Ordinary LoRA SFT} \\\\ \text{PG-LLM} \\\\ \text{PLD} \\\\ \text{DFlash} \\\\ \text{PG-LLM+DFlash} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 72.30 & 100.00 & 1.00\times \\\\ 71.36 & 62.62 & 1.21\times \\\\ 72.30 & 29.64 & 1.76\times \\\\ 72.30 & 15.62 & 2.24\times \\\\ 71.36 & 17.45 & \mathbf{2.64\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 55.24 & 100.00 & 1.00\times \\\\ 56.19 & 64.88 & 1.28\times \\\\ 55.24 & 21.83 & 1.79\times \\\\ 55.24 & 14.68 & 2.18\times \\\\ 56.19 & 16.24 & \mathbf{2.57\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 60.82 & 100.00 & 1.00\times \\\\ 62.89 & 83.21 & 1.12\times \\\\ 60.82 & 41.72 & 1.67\times \\\\ 60.82 & 16.13 & 2.12\times \\\\ 62.89 & 18.50 & \mathbf{2.31\times} \end{array} \end{array}$$ **Table 1b. Comparison across decoding methods on code editing benchmarks with Qwen3-30B-A3B-Instruct-2507** $$\begin{array}{l|c|c|c} & \text{CommitPackFT} & \text{CanItEdit} & \text{EditEval} \\\\ \hline \begin{array}{l} \text{Method} \\\\ \hline \text{Ordinary LoRA SFT} \\\\ \text{PG-LLM} \\\\ \text{PLD} \\\\ \text{DFlash} \\\\ \text{PG-LLM+DFlash} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 83.57 & 100.00 & 1.00\times \\\\ 83.10 & 64.83 & 1.18\times \\\\ 83.57 & 29.71 & 1.74\times \\\\ 83.57 & 14.71 & 2.11\times \\\\ 83.10 & 16.25 & \mathbf{2.48\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 66.67 & 100.00 & 1.00\times \\\\ 66.67 & 66.38 & 1.26\times \\\\ 66.67 & 23.68 & 1.83\times \\\\ 66.67 & 14.29 & 2.08\times \\\\ 66.67 & 17.23 & \mathbf{2.49\times} \end{array} & \begin{array}{rrr} \text{Acc}\uparrow & \text{Token}\downarrow & \text{Speedup}\uparrow \\\\ \hline 73.71 & 100.00 & 1.00\times \\\\ 75.26 & 82.42 & 1.13\times \\\\ 73.71 & 41.57 & 1.62\times \\\\ 73.71 & 15.38 & 2.04\times \\\\ 75.26 & 19.74 & \mathbf{2.16\times} \end{array} \end{array}$$ (continue below)
AuthorsOpenReview ↗

Response to Reviewer mQ6Q (2/2)

(continue Weakness 1 and Question 1 above)

In Table 1, the token ratios for DFlash and PG-LLM+DFlash measure the number of tokens directly generated by the target model relative to the ordinary LoRA baseline’s output length. PG-LLM’s specialized training may introduce a mismatch between the drafter and the target model’s prediction distributions, leading to more draft rejections and a slightly higher target-generated token ratio. Nevertheless, PG-LLM can directly copy selected source spans, bypassing neural drafting for those spans. This reduces the amount of content that DFlash’s shallow draft model needs to generate, enabling additional acceleration beyond standalone speculative decoding.

Overall, the experimental results show that PG-LLM, as a learned context-reuse method, can be combined with existing speculative decoding methods to achieve greater speedups.


Weakness 2 and Question 2: Training controls and paired correctness.

We have added a generation-only LoRA SFT control using the same 22,614 training examples, with the full reference output as its target and no pointer head. Tables 1a–1b report all six benchmarks and also retain the original PG-LLM SFT results to distinguish the training stages.

The control experiments show that PG-LLM achieves code editing and code generation performance comparable to a conventional model fine-tuned on the same SFT data. Our main contribution is a learned pointer-generator mechanism that identifies and directly reuses relevant spans from the input context, accelerating decoding while preserving comparable overall output quality.

Table 1a. Comparison of code editing and generation performance across training settings with Qwen3-4B-Instruct-2507

Model / trainingCommitPackFT ↑CanItEdit ↑EditEval ↑LiveCodeBench ↑BigCodeBench-Hard ↑MBPP ↑
Original model70.4252.3860.3125.1922.9778.04
Ordinary LoRA SFT72.3055.2460.8223.6622.3077.25
PG-LLM SFT71.8354.2957.2221.3718.2474.60
PG-LLM SFT+DPO71.3656.1962.8924.4323.6579.10

Table 1b. Comparison of code editing and generation performance across training settings with Qwen3-30B-A3B-Instruct-2507

Model / trainingCommitPackFT ↑CanItEdit ↑EditEval ↑LiveCodeBench ↑BigCodeBench-Hard ↑MBPP ↑
Original model82.1664.7673.2043.5141.2288.89
Ordinary LoRA SFT83.5766.6773.7142.7540.5488.10
PG-LLM SFT83.5767.6270.6238.9337.1685.71
PG-LLM SFT+DPO83.1066.6775.2641.9842.5788.36

Weakness 3 and Question 3: Repeated-span guard, timed-output correctness, and timing variation.

We checked the actual timed EditEval outputs. For both PG-LLM 4B and 30B, exactly one of 194 requests (0.52%) triggered the repeated-span guard in each of the three runs. Functional accuracy was 62.89% for 4B and 75.26% for 30B in every run, corresponding to 122 and 146 correct responses. All 194 requests, including the guard-triggered request, remain in the evaluation denominator. The corresponding PG-LLM EditEval results in the submitted Table 1 were obtained with the same vLLM engine, guard, and evaluation settings.

Table 1. Guard activations and functional accuracy across three EditEval runs

ModelGuard-triggered requests ↓Correct responses ↑Functional accuracy ↑
PG-LLM 4B1 / 1 / 1122/19462.89%
PG-LLM 30B1 / 1 / 1146/19475.26%

Table 2. EditEval throughput and variation across three runs

Model and implementationThroughput ↑ (req/s)
4B vLLM base9.63 ± 0.42
4B vLLM PG-LLM10.79 ± 0.26
30B vLLM base5.73 ± 0.15
30B vLLM PG-LLM6.47 ± 0.02

Question 4: Configuration details.

The pointer dimension equals the backbone hidden dimension: 2,560 for 4B and 2,048 for 30B. AST-aligned copy candidates must contain at least 32 characters and 4 tokens. Reconstructed SFT sequences are filtered at 2,048 tokens; the longest retained sequence is 2,043 tokens, as reported in Appendix C.1. This SFT filtering limit is separate from the 4,096-token limit used for 4B DPO.

For NextCoder preprocessing, we retain examples with source length 250–1,500 tokens, 4–20 total actions, 2–10 copy actions, at most 10 generation actions, character-level copy ratio 0.50–0.85, at most 350 LM-supervised tokens, and edit intensity 0.02–0.50. The action-count thresholds apply before AST reconstruction, which may change the number of actions.

The benchmark decoding budgets are listed below. Appendices A–D of the submitted PDF describe the architecture, training, and evaluation subsets.

Table 1. Output token budgets for the six code benchmarks

BenchmarkOutput-token budget
CommitPackFT1,024
CanItEdit4,096
EditEval1,024
MBPP+1,024
LiveCodeBench2,048
BigCodeBench-Hard1,536

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.