- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a fill in the middle (FIM) framework for coding LLMs, which uses a sophisticated context selection method: by selecting the relevant chunks by their combined utility, we can increase the generator exact match (EM) scores in FIM completions. The authors create a dataset using combination of differnet context, minimizing frozen generator's cross-entropy loss and train a context reranker model to select such chunks and also train the generator model via SFT. The authors additionally introduce a benchmark, RealBench-FIM and evaluate their system (reranker + decoder) on this benchmark, outperforming the baselines of much larger sizes.
Critical strengths
The method seems to work well, the paper is easy to follow and the method, to the best of my knowledge, is novel. The results are great: a system of 7b model and 0.6b reranker models outperform much larger models in FIM tasks on Verilog and SystemVerilog code.
Critical weaknesses
Only one generator model is evaluated: qwen 2.5 7b. This model is specially trained for FIM (as evidenced by FIM special tokens in the tokenizer), with no other generator model ablation to suggest whether this approach generalizes. Same thing for reranker: only Jina is evaluated, maybe this approach depends on the reranker pretraining and will not generalize.
Scaling laws are not explored: 47647 training examples is quite a lot and having information whether the reranker is undertrained, overtrained or trained perfectly will be very useful for reproduction of this paper.
Selected larger models, such as DeepSeek 3.2, GPT-5.4, GLM-5.2 and Claude Opus 4.6 were optimized for agentic work instead of FIM completion, which might make them weaker on the task. The baselines should be specialized FIM models, such as Qwen3-Next-Coder, StarCoder or Codestral.
The performance of the full pipeline depends on SFT and reranker, and with the frozen generator the trained reranker improves the scores by 0.83%, compared to 27% improvement from the full pipeline. Therefore it is unclear, whether the reranker works or not, I suggest ablating this with SFT'd generator model and a single chunk utility reranker.
Additionally, the benchmark is made from only Verilog and SystemVerilog code, which is a pretty unique domain. I wonder whether this percieved quality will remain on more popular coding languages, such as Python, C++ or Rust.
The benchmark itself is rather small, consisting only of 145 examples. This is way too little, without any statistical significance testing we cannot say whether the approach actually works. The main methodological novelty gives 0.83% of the score and it seems like a noise on this sample size.
Questions for authors
Will the lead hold when comparing your approach to specialized FIM models instead of general LLMs?
Does the approach generalize to other reranker and generator models?
Does the approach generalize to languages other than Verilog and SystemVerilog? Will your approach win other with the same lead or it is specific for these two long tail languages?
What are the scaling laws for the reranker training?
Will the scores hold without finetuned reranker? Is it necessary for the final model performance?
What is the 95% CI values for benchmarks? Is the performance increase due to reranker training statistically significant?
4 comments
W4/Q5:
The performance of the full pipeline depends on SFT and reranker, and with the frozen generator the trained reranker improves the scores by 0.83%, compared to 27% improvement from the full pipeline. Therefore it is unclear, whether the reranker works or not, I suggest ablating this with SFT'ed generator model and a single chunk utility reranker.
Will the scores hold without finetuned reranker? Is it necessary for the final model performance?
To answer this question, we conducted new experiments on different reranker and SFT methods. It is shown that CoRTL with combination utility reranker outperformed all other settings listed in the Table below, including no reranker, single-chunk utility reranker, and pretrained Jina with combination utility SFT. The full-system ablation appears in Section 4.4 and Table 3. Appendix A.9 adds the comparison with pretrained Jina using the same fine-tuned generator.
| Setup | RealBench-FIM func@1 | RealBench-FIM func@5 |
|---|---|---|
| No retrieval | 67.00% | 69.43% |
| Single-chunk utility reranker + single-chunk utility SFT | 74.55% | 75.97% |
| Pretrained Jina + combination utility SFT | 74.55% | 75.58% |
| Combination utility reranker + combination utility SFT (CoRTL) | 75.31% | 76.36% |
W5/Q3:
Additionally, the benchmark is made from only Verilog and SystemVerilog code, which is a pretty unique domain. I wonder whether this perceived quality will remain on more popular coding languages, such as Python, C++ or Rust.
Does the approach generalize to languages other than Verilog and SystemVerilog? Will your approach win other with the same lead or it is specific for those two long tail languages?
Due to the dataset constraints and time limit, we conducted a proof-of-concept experiment to show that our combination utility search may generalize to Python code completion.
Our method include two stages: stage 1 is supervision construction via combination utility search, while stage 2 is reranker and completion generator training. For the proof-of-concept experiment, we only evaluate the first stage, utility-based context selection with a fixed generator (Qwen2.5-Coder-7B-Instruct). We select single-chunk utility selection as the baseline and test whether combination utility search performs better than it under the same budget. We choose EM (exact-match) and ES (edit similarity) as our metric and compute confidence intervals for them. We use RepoEval [1], a benchmark for repository-level Python completion, testing line and API completion.
On RepoEval, our combination utility search achieves 64.72% exact match (EM) and 82.72% edit similarity (ES), outperforming single-chunk utility selection (62.88% EM and 81.85% ES). The paired 95% confidence intervals for the EM and ES gains are [1.12%, 2.56%] and [0.43%, 1.31%], respectively. Both intervals exclude zero, indicating statistically significant improvements. The new RepoEval comparison is in Appendix A.9 (Table 14).
| RepoEval task | EM (single-chunk utility) | EM (combination utility) | ES (single-chunk utility) | ES (combination utility) |
|---|---|---|---|---|
| Line | 67.06% | 68.38% | 83.09% | 83.71% |
| API | 58.69% | 61.06% | 80.62% | 81.73% |
| Overall | 62.88% | 64.72% | 81.85% | 82.72% |
[1] Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., ... & Chen, W. (2023, December). Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 2471-2484).
W6/Q6:
The benchmark itself is rather small, consisting only of 145 examples. This is way too little, without any statistical significance testing we cannot say whether the approach actually works. The main methodological novelty gives 0.83% of the score and it seems like a noise on this sample size.
What is the 95% CI values for benchmarks? Is the performance increase due to reranker training statistically significant?
To answer this question, we used paired bootstrap resampling to assess the gains from the full CoRTL system and from reranker training: CoRTL outperformed RLCoder and RepoShapley on both benchmarks' primary metrics and scored higher than GPT-5.4 in the original evaluation setting, and the trained reranker improved RealBench-FIM func@1 over pretrained Jina with the original generator fixed. CoRTL reaches 75.31% func@1 on RealBench-FIM (95% CI [68.38%, 82.07%]) and 66.78% EM on RTL-Repo (95% CI [64.05%, 69.51%]). The gains over GPT-5.4 are 6.34% on RealBench-FIM and 4.17% on RTL-Repo. With the original generator fixed, the trained reranker improves RealBench-FIM func@1 over pretrained Jina by 7.14% (paired 95% CI [3.34, 11.31]). In the same setting, combination utility supervision also improved func@1 over single-chunk utility supervision (+0.83%; paired 95% CI [−2.62, 4.31]). The tables report paired 95% CIs from 20,000 resamples of the completion targets. Appendix A.8 and Table 10 now give per-method confidence intervals for both benchmarks.
Full-system ablations on RealBench-FIM
| Comparison | func@1 (%) [95% CI] | Gain (pp) | Paired 95% CI (pp) |
|---|---|---|---|
| CoRTL vs no retrieval | 75.31 [68.38, 82.07] vs. 67.00 [59.45, 74.31] | +8.31 | [2.66, 14.24] |
| CoRTL vs single-chunk utility | 75.31 [68.38, 82.07] vs. 74.55 [67.55, 81.24] | +0.76 | [−2.79, 4.55] |
Main comparisons on RealBench-FIM
| Comparison | func@1 (%) [95% CI] | Gain (pp) | Paired 95% CI (pp) |
|---|---|---|---|
| CoRTL vs RepoShapley | 75.31 [68.38, 82.07] vs. 61.97 [54.55, 69.34] | +13.34 | [7.48, 19.55] |
| CoRTL vs RLCoder | 75.31 [68.38, 82.07] vs. 44.48 [37.21, 51.86] | +30.83 | [+23.83, +38.00] |
| CoRTL vs Claude Opus 4.6 | 75.31 [68.38, 82.07] vs. 81.38 [75.17, 87.59] | −6.07 | [−13.72, +1.48] |
| CoRTL vs GPT-5.4 | 75.31 [68.38, 82.07] vs. 68.97 [61.38, 75.86] | +6.34 | [−0.76, +13.83] |
Paired comparisons on RTL-Repo
| Comparison | EM (%) [95% CI] | EM gain (pp) [Paired 95% CI] | ES (%) [95% CI] | ES gain (pp) [Paired 95% CI] |
|---|---|---|---|---|
| CoRTL vs RepoShapley | 66.78 [64.05, 69.51] vs. 61.41 [58.60, 64.22] | +5.37 [+2.81, +8.01] | 89.03 [87.78, 90.25] vs. 88.79 [87.61, 89.93] | +0.24 [−0.94, +1.42] |
| CoRTL vs RLCoder | 66.78 [64.05, 69.51] vs. 62.10 [59.28, 64.91] | +4.69 [+2.39, +7.07] | 89.03 [87.78, 90.25] vs. 86.66 [85.23, 88.04] | +2.37 [+1.17, +3.60] |
| CoRTL vs Claude Opus 4.6 | 66.78 [64.05, 69.51] vs. 70.87 [68.23, 73.42] | −4.09 [−6.81, −1.28] | 89.03 [87.78, 90.25] vs. 82.41 [81.25, 83.56] | +6.62 [+5.24, +8.00] |
| CoRTL vs GPT-5.4 | 66.78 [64.05, 69.51] vs. 62.61 [59.88, 65.42] | +4.17 [+1.36, +6.99] | 89.03 [87.78, 90.25] vs. 86.33 [85.06, 87.58] | +2.70 [+1.33, +4.09] |
Thank you for the additional experiments. However, the improvement of CoRTL over single-chunk utility is not statistically significant (95% CI [−2.79, 4.55]). Similarly, the improvement from combination utility supervision (+0.83 pp) is not statistically significant (95% CI [−2.62, 4.31]).
Since combination utility is the central methodological contribution of the paper, I do not think the presented evidence is sufficient to establish its effectiveness over the simpler single-chunk utility baseline.
We thank the reviewer for recognizing the novelty of combination utility search and the results achieved with a compact RTL completion system. We added specialized FIM baselines, evaluated a second generator and reranker, and estimated confidence intervals on both benchmarks to clarify where the gains come from.
W1/Q2:
Our approach generalizes to other generator and reranker models: in two new experiments on RealBench-FIM, we separately switched to a Llama-3.1-8B-Instruct generator and a Qwen3-Reranker-0.6B reranker, and combination utility yielded higher func@1 and func@5 than single-chunk utility in both settings.
Appendix A.9 describes the Llama generator and Qwen3 reranker experiments, with results in Tables 11 and 12.
W2/Q4:
On RealBench-FIM, both metrics improve as we increase the reranker training data, reaching their highest values with the full dataset. We kept the original generator and candidate pool fixed and trained on the same setting as our paper. Table 13 summarizes the training data size ablation added to Appendix A.9.
W3/Q1:
Following your advice, we added three specialized FIM baselines on RealBench-FIM, and CoRTL outperformed all three with 75.31% func@1, 22.00% ahead of the strongest baseline, StarCoderBase-7B. We evaluated each baseline on the same 145 tasks, providing the target file's prefix and suffix in the model's native FIM format. We revised Section 4.1 and Table 1 to include the specialized FIM baselines. Appendix A.6 gives their evaluation settings.