acceptodds
Under review as a conference paper at ICLR 2027

CoRTL: Combination Utility Guided Context Retrieval for Repository-Level RTL Code Completion

Abstract

Completing register-transfer level (RTL) code within repositories often requires declarations distributed across multiple chunks, making context selection sensitive to interactions among retrieved chunks. We show that selecting chunks by individual utility frequently misses higher-utility combinations. We introduce CoRTL, a fill-in-the-middle (FIM) completion framework that learns from context combinations. Our method, combination utility search, uses a fixed generator to score combinations by their reduction in mean per-token cross-entropy on the reference target code relative to in-file context alone. The selected combinations provide candidate-level ranking supervision for a context reranker and cross-file context for fine-tuning a completion generator to predict the reference target code. We also construct 47,647 FIM training examples and introduce RealBench-FIM, a repository-level RTL FIM benchmark with testbench-based functional correctness evaluation. With a 0.6B reranker and a 7B completion generator, CoRTL achieves 75.31% func@1 on RealBench-FIM and 66.78% exact match on RTL-Repo, outperforming RLCoder and RepoShapley and exceeding GPT-5.4 by 6.34% and 4.17%, respectively, without using a larger teacher model for supervision construction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Code Completion, Verilog

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WoqZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a fill in the middle (FIM) framework for coding LLMs, which uses a sophisticated context selection method: by selecting the relevant chunks by their combined utility, we can increase the generator exact match (EM) scores in FIM completions. The authors create a dataset using combination of differnet context, minimizing frozen generator's cross-entropy loss and train a context reranker model to select such chunks and also train the generator model via SFT. The authors additionally introduce a benchmark, RealBench-FIM and evaluate their system (reranker + decoder) on this benchmark, outperforming the baselines of much larger sizes.

Critical strengths

The method seems to work well, the paper is easy to follow and the method, to the best of my knowledge, is novel. The results are great: a system of 7b model and 0.6b reranker models outperform much larger models in FIM tasks on Verilog and SystemVerilog code.

Critical weaknesses

Only one generator model is evaluated: qwen 2.5 7b. This model is specially trained for FIM (as evidenced by FIM special tokens in the tokenizer), with no other generator model ablation to suggest whether this approach generalizes. Same thing for reranker: only Jina is evaluated, maybe this approach depends on the reranker pretraining and will not generalize.

Scaling laws are not explored: 47647 training examples is quite a lot and having information whether the reranker is undertrained, overtrained or trained perfectly will be very useful for reproduction of this paper.

Selected larger models, such as DeepSeek 3.2, GPT-5.4, GLM-5.2 and Claude Opus 4.6 were optimized for agentic work instead of FIM completion, which might make them weaker on the task. The baselines should be specialized FIM models, such as Qwen3-Next-Coder, StarCoder or Codestral.

The performance of the full pipeline depends on SFT and reranker, and with the frozen generator the trained reranker improves the scores by 0.83%, compared to 27% improvement from the full pipeline. Therefore it is unclear, whether the reranker works or not, I suggest ablating this with SFT'd generator model and a single chunk utility reranker.

Additionally, the benchmark is made from only Verilog and SystemVerilog code, which is a pretty unique domain. I wonder whether this percieved quality will remain on more popular coding languages, such as Python, C++ or Rust.

The benchmark itself is rather small, consisting only of 145 examples. This is way too little, without any statistical significance testing we cannot say whether the approach actually works. The main methodological novelty gives 0.83% of the score and it seems like a noise on this sample size.

Questions for authors

Will the lead hold when comparing your approach to specialized FIM models instead of general LLMs?

Does the approach generalize to other reranker and generator models?

Does the approach generalize to languages other than Verilog and SystemVerilog? Will your approach win other with the same lead or it is specific for these two long tail languages?

What are the scaling laws for the reranker training?

Will the scores hold without finetuned reranker? Is it necessary for the final model performance?

What is the 95% CI values for benchmarks? Is the performance increase due to reranker training statistically significant?

4 comments

We thank the reviewer for recognizing the novelty of combination utility search and the results achieved with a compact RTL completion system. We added specialized FIM baselines, evaluated a second generator and reranker, and estimated confidence intervals on both benchmarks to clarify where the gains come from.

W1/Q2:

Only one generator model is evaluated: qwen 2.5 7b. This model is specially trained for FIM (as evidenced by FIM special tokens in the tokenizer), with no other generator model ablation to suggest whether this approach generalizes. Same thing for reranker: only jina is evaluated, maybe this approach depends on the reranker pretraining and will not generalize.

Does the approach generalize to other reranker and generator models?

Our approach generalizes to other generator and reranker models: in two new experiments on RealBench-FIM, we separately switched to a Llama-3.1-8B-Instruct generator and a Qwen3-Reranker-0.6B reranker, and combination utility yielded higher func@1 and func@5 than single-chunk utility in both settings.

  1. With the same Jina-reranked contexts at inference, both Llama variants improved over the original checkpoint after two epochs of SFT. The variant trained on contexts from combination utility search reached 70.34% func@1, versus 68.38% with single-chunk utility selection. This shows that the combinations found by the Qwen teacher also benefit a different generator.
  2. For Qwen3-Reranker-0.6B, we kept the BM25 + UniXcoder candidate pool, the top-10 context budget, and the same fine-tuned Qwen2.5-Coder-7B-Instruct generator fixed. Combination utility supervision reached 75.14% func@1, ahead of both the pretrained reranker and the reranker trained with single-chunk utility. This shows that combination utility supervision also works with a reranker initialized from Qwen3.

Appendix A.9 describes the Llama generator and Qwen3 reranker experiments, with results in Tables 11 and 12.

Llama-3.1-8B-Instruct generatorfunc@1func@5
Original checkpoint15.52%20.05%
SFT on contexts from single-chunk utility selection68.38%71.09%
SFT on contexts from combination utility search70.34%72.73%
Qwen3-Reranker-0.6Bfunc@1func@5
Pretrained reranker74.31%75.74%
Single-chunk utility supervision73.48%75.31%
Combination utility supervision75.14%76.24%

W2/Q4:

Scaling laws are not explored: 47647 training examples is quite a lot and having information whether the reranker is undertrained, overtrained or trained perfectly will be very useful for reproduction of this paper.

What are the scaling laws for the reranker training?

On RealBench-FIM, both metrics improve as we increase the reranker training data, reaching their highest values with the full dataset. We kept the original generator and candidate pool fixed and trained on the same setting as our paper. Table 13 summarizes the training data size ablation added to Appendix A.9.

Reranker training dataTraining examplesfunc@1func@5
0% (pretrained Jina)041.17%49.17%
20%9,08045.86%54.71%
60%27,24146.90%56.20%
100% (checkpoint used in submission)45,40248.31%57.65%

W3/Q1:

Selected larger models, such as DeepSeek 3.2, GPT-5.4, GLM-5.2 and Claude Opus 4.6 were optimized for agentic work instead of FIM completion, which might make them weaker on the task. The baselines should be specialized FIM models, such as Qwen3-Next-Coder, StarCoder or Codestral.

Will the lead hold when comparing your approach to specialized FIM models instead of general LLMs?

Following your advice, we added three specialized FIM baselines on RealBench-FIM, and CoRTL outperformed all three with 75.31% func@1, 22.00% ahead of the strongest baseline, StarCoderBase-7B. We evaluated each baseline on the same 145 tasks, providing the target file's prefix and suffix in the model's native FIM format. We revised Section 4.1 and Table 1 to include the specialized FIM baselines. Appendix A.6 gives their evaluation settings.

Methodfunc@1func@5
StarCoderBase-7B53.31%61.38%
StarCoder2-15B35.41%41.03%
Qwen3-Coder-30B-A3B-Instruct38.83%40.61%
CoRTL-7B75.31%76.36%

W4/Q5:

The performance of the full pipeline depends on SFT and reranker, and with the frozen generator the trained reranker improves the scores by 0.83%, compared to 27% improvement from the full pipeline. Therefore it is unclear, whether the reranker works or not, I suggest ablating this with SFT'ed generator model and a single chunk utility reranker.

Will the scores hold without finetuned reranker? Is it necessary for the final model performance?

To answer this question, we conducted new experiments on different reranker and SFT methods. It is shown that CoRTL with combination utility reranker outperformed all other settings listed in the Table below, including no reranker, single-chunk utility reranker, and pretrained Jina with combination utility SFT. The full-system ablation appears in Section 4.4 and Table 3. Appendix A.9 adds the comparison with pretrained Jina using the same fine-tuned generator.

SetupRealBench-FIM func@1RealBench-FIM func@5
No retrieval67.00%69.43%
Single-chunk utility reranker + single-chunk utility SFT74.55%75.97%
Pretrained Jina + combination utility SFT74.55%75.58%
Combination utility reranker + combination utility SFT (CoRTL)75.31%76.36%

W5/Q3:

Additionally, the benchmark is made from only Verilog and SystemVerilog code, which is a pretty unique domain. I wonder whether this perceived quality will remain on more popular coding languages, such as Python, C++ or Rust.

Does the approach generalize to languages other than Verilog and SystemVerilog? Will your approach win other with the same lead or it is specific for those two long tail languages?

Due to the dataset constraints and time limit, we conducted a proof-of-concept experiment to show that our combination utility search may generalize to Python code completion.

Our method include two stages: stage 1 is supervision construction via combination utility search, while stage 2 is reranker and completion generator training. For the proof-of-concept experiment, we only evaluate the first stage, utility-based context selection with a fixed generator (Qwen2.5-Coder-7B-Instruct). We select single-chunk utility selection as the baseline and test whether combination utility search performs better than it under the same budget. We choose EM (exact-match) and ES (edit similarity) as our metric and compute confidence intervals for them. We use RepoEval [1], a benchmark for repository-level Python completion, testing line and API completion.

On RepoEval, our combination utility search achieves 64.72% exact match (EM) and 82.72% edit similarity (ES), outperforming single-chunk utility selection (62.88% EM and 81.85% ES). The paired 95% confidence intervals for the EM and ES gains are [1.12%, 2.56%] and [0.43%, 1.31%], respectively. Both intervals exclude zero, indicating statistically significant improvements. The new RepoEval comparison is in Appendix A.9 (Table 14).

RepoEval taskEM (single-chunk utility)EM (combination utility)ES (single-chunk utility)ES (combination utility)
Line67.06%68.38%83.09%83.71%
API58.69%61.06%80.62%81.73%
Overall62.88%64.72%81.85%82.72%

[1] Zhang, F., Chen, B., Zhang, Y., Keung, J., Liu, J., Zan, D., ... & Chen, W. (2023, December). Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 2471-2484).

W6/Q6:

The benchmark itself is rather small, consisting only of 145 examples. This is way too little, without any statistical significance testing we cannot say whether the approach actually works. The main methodological novelty gives 0.83% of the score and it seems like a noise on this sample size.

What is the 95% CI values for benchmarks? Is the performance increase due to reranker training statistically significant?

To answer this question, we used paired bootstrap resampling to assess the gains from the full CoRTL system and from reranker training: CoRTL outperformed RLCoder and RepoShapley on both benchmarks' primary metrics and scored higher than GPT-5.4 in the original evaluation setting, and the trained reranker improved RealBench-FIM func@1 over pretrained Jina with the original generator fixed. CoRTL reaches 75.31% func@1 on RealBench-FIM (95% CI [68.38%, 82.07%]) and 66.78% EM on RTL-Repo (95% CI [64.05%, 69.51%]). The gains over GPT-5.4 are 6.34% on RealBench-FIM and 4.17% on RTL-Repo. With the original generator fixed, the trained reranker improves RealBench-FIM func@1 over pretrained Jina by 7.14% (paired 95% CI [3.34, 11.31]). In the same setting, combination utility supervision also improved func@1 over single-chunk utility supervision (+0.83%; paired 95% CI [−2.62, 4.31]). The tables report paired 95% CIs from 20,000 resamples of the completion targets. Appendix A.8 and Table 10 now give per-method confidence intervals for both benchmarks.

Full-system ablations on RealBench-FIM

Comparisonfunc@1 (%) [95% CI]Gain (pp)Paired 95% CI (pp)
CoRTL vs no retrieval75.31 [68.38, 82.07] vs. 67.00 [59.45, 74.31]+8.31[2.66, 14.24]
CoRTL vs single-chunk utility75.31 [68.38, 82.07] vs. 74.55 [67.55, 81.24]+0.76[−2.79, 4.55]

Main comparisons on RealBench-FIM

Comparisonfunc@1 (%) [95% CI]Gain (pp)Paired 95% CI (pp)
CoRTL vs RepoShapley75.31 [68.38, 82.07] vs. 61.97 [54.55, 69.34]+13.34[7.48, 19.55]
CoRTL vs RLCoder75.31 [68.38, 82.07] vs. 44.48 [37.21, 51.86]+30.83[+23.83, +38.00]
CoRTL vs Claude Opus 4.675.31 [68.38, 82.07] vs. 81.38 [75.17, 87.59]−6.07[−13.72, +1.48]
CoRTL vs GPT-5.475.31 [68.38, 82.07] vs. 68.97 [61.38, 75.86]+6.34[−0.76, +13.83]

Paired comparisons on RTL-Repo

ComparisonEM (%) [95% CI]EM gain (pp) [Paired 95% CI]ES (%) [95% CI]ES gain (pp) [Paired 95% CI]
CoRTL vs RepoShapley66.78 [64.05, 69.51] vs. 61.41 [58.60, 64.22]+5.37 [+2.81, +8.01]89.03 [87.78, 90.25] vs. 88.79 [87.61, 89.93]+0.24 [−0.94, +1.42]
CoRTL vs RLCoder66.78 [64.05, 69.51] vs. 62.10 [59.28, 64.91]+4.69 [+2.39, +7.07]89.03 [87.78, 90.25] vs. 86.66 [85.23, 88.04]+2.37 [+1.17, +3.60]
CoRTL vs Claude Opus 4.666.78 [64.05, 69.51] vs. 70.87 [68.23, 73.42]−4.09 [−6.81, −1.28]89.03 [87.78, 90.25] vs. 82.41 [81.25, 83.56]+6.62 [+5.24, +8.00]
CoRTL vs GPT-5.466.78 [64.05, 69.51] vs. 62.61 [59.88, 65.42]+4.17 [+1.36, +6.99]89.03 [87.78, 90.25] vs. 86.33 [85.06, 87.58]+2.70 [+1.33, +4.09]

Thank you for the additional experiments. However, the improvement of CoRTL over single-chunk utility is not statistically significant (95% CI [−2.79, 4.55]). Similarly, the improvement from combination utility supervision (+0.83 pp) is not statistically significant (95% CI [−2.62, 4.31]).

Since combination utility is the central methodological contribution of the paper, I do not think the presented evidence is sufficient to establish its effectiveness over the simpler single-chunk utility baseline.

Reviewer_nCmLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In coding scenarios, how can LLMs be applied to RTL code generation and completion based on cross-file references? The paper presents CoRTL, a retrieval-augmented framework for repository-level RTL fill-in-the-middle completion. Its central premise is that individually useful code chunks may not form the most useful context combination. CoRTL uses a fixed generator’s reduction in reference-target cross-entropy to guide beam search over chunk combinations. The selected combinations supervise a context reranker and provide training contexts for generator fine-tuning, whose inference uses retrieval, reranking, and completion without combination search. They also construct a repository-level RTL training corpus with 47,647 FIM examples and introduce RealBench-FIM. With a 0.6B reranker and a 7B completion generator, CoRTL outperforms the evaluated RTL-domain models and repository-level frameworks on both benchmarks, with absolute margins over GPT-5.4 of 6.34% in overall func@1 on RealBench-FIM (only 145 tasks from 3 projects, with no OOD evaluation) and a slight improvement in RTL-Repo.

Critical strengths

Although commercial coding agents are powerful, the method proposed in this paper is a good solution for scenarios requiring offline deployment, as it only requires a 0.6B reranker model and Qwen2.5-Coder-7B. It also shows notable improvements over other small open-weight models (at least 8%), which is the most important aspect, since incorrect garbage code wastes even more time.

The paper also provides training data and a benchmark with 145 tasks for multi-granularity RTL code completion.

Critical weaknesses

The paper does not compare completion speed, which is important because if the workflow is too slow, it loses its practical advantage. This workflow definitely has advantages over some commercial models, as GPT-series models can be a bit slow. However, it is less clear when compared with the latest Deepseek-Flash, which can generate around 100 tokens/s. At the same time, the paper does not evaluate the speed and accuracy of commercial models when used with a harness (such as Codex or dsh). In fact, these harnesses already include a RAG-like workflow for code retrieval.

The number of evaluation benchmarks is limited, and the self-constructed benchmark contains only 145 tasks. The accuracy may be overestimated because the code styles or data distributions of the training and completion tasks are similar. Also, the way code is masked to construct the dataset may not match real-world coding scenarios, which could also affect the practical value of the method and dataset.

Questions for authors

How do different methods compare in terms of completion speed? Can the latest deepseek models achieve both high speed and accuracy with a medium thinking budget? And how do commercial models perform when paired with a harness?

Are there other OOD datasets that can be used to evaluate the accuracy of the method? The results on RTL-Repo are not very promising.

3 comments

We thank the reviewer for highlighting CoRTL's suitability for offline deployment and the value of our RTL completion dataset. On RealBench-FIM, CoRTL completes tasks faster than both DeepSeek Flash and Codex with GPT-5.4, with similar func@1. On RTL-Repo, CoRTL's gains over both retrieval baselines are statistically significant.

W1/Q1:

The paper does not compare completion speed, which is important because if the workflow is too slow, it loses its practical advantage. This workflow definitely has advantages over some commercial models, as GPT-series models can be a bit slow. However, it is less clear when compared with the latest Deepseek-Flash, which can generate around 100 tokens/s.

How do different methods compare in terms of completion speed? Can the latest deepseek models achieve both high speed and accuracy with a medium thinking budget?

To answer your question, we measured completion time on the same 145 RealBench-FIM tasks and found that DeepSeek-V4.1-Flash (released on 2026.10.17, far behind our submission) with a medium thinking budget scored only 0.55% higher in func@1 and took about 8.9 times as long as CoRTL. CoRTL averages 1.30 s per target: 0.050 s for candidate retrieval, 0.257 s for reranking, 0.035 s for prompt construction, and 0.953 s for generation. Flash averages 11.58 s per API request. The new Section 4.5 compares completion accuracy and latency in Figure 4(b–c). Appendix A.7 explains how we measured time.

Methodfunc@1Mean measured time (s/task)
DeepSeek-V4.1-Flash (medium thinking)75.86%11.58
CoRTL75.31%1.30

W2/Q2:

At the same time, the paper does not evaluate the speed and accuracy of commercial models when used with a harness (such as Codex or dsh). In fact, these harnesses already include a RAG-like workflow for code retrieval.

And how do commercial models perform when paired with a harness?

We paired GPT-5.4 with Codex CLI for a new evaluation on all 145 RealBench-FIM tasks, where it scored 3.31% above CoRTL in func@1 and took about 32 times as long on average. Codex could search the project's RTL sources and edit the target file directly. We evaluated the resulting full files with the original RealBench verifier. Section 4.5 and Figure 4(b) now include the Codex CLI results. The harness setup is described in Appendix A.7.

Methodfunc@1Mean measured time (s/task)
Codex CLI with GPT-5.478.62%41.42
CoRTL75.31%1.30

W3/Q3:

The number of evaluation benchmarks is limited, and the self-constructed benchmark contains only 145 tasks. The accuracy may be overestimated because the code styles or data distributions of the training and completion tasks are similar.

Are there other OOD datasets that can be used to evaluate the accuracy of the method? The results on RTL-Repo are not very promising.

Our training data is fully decontaminated. To prevent overlap with the evaluation data, we screened the training repositories against both RTL benchmarks during training data construction, using normalized token containment and shingle similarity, and excluded any matching repository (Appendix A.1).

Our results on RTL-Repo is statistically positive. We further estimated 95% confidence intervals on RTL-Repo, using paired bootstrap resampling over 1,174 completion targets. CoRTL achieved 66.78% exact match (95% CI [64.05%, 69.51%]), outperforming each of the five baselines in the paired comparisons (see table below). The new confidence interval analysis in Appendix A.8 includes RTL-Repo results for each method (Table 10).

MethodRTL-Repo EMCoRTL gain (pp)Paired 95% CI (pp)
CoRTL66.78%——
RLCoder62.10%+4.69[2.39, 7.07]
RepoShapley61.41%+5.37[2.81, 8.01]
GPT-5.462.61%+4.17[1.36, 6.99]
DeepSeek-V3.252.04%+14.74[11.75, 17.80]
GLM-5.259.71%+7.07[4.26, 9.97]

W4:

Also, the way code is masked to construct the dataset may not match real-world coding scenarios, which could also affect the practical value of the method and dataset.

RealBench-FIM's masking reflects real-world local RTL edits: the model completes a line or a code construct within an existing module while the surrounding code and interfaces remain in place. We use Tree-sitter to set the boundaries of instantiation-and-call and structured-span targets. With the rest of the design intact, we can insert the completion back into the module and test it with the original dependencies and testbench (Appendix A.1). To support this, prior benchmarks on other coding languages also use syntactic structure to construct completion tasks: SAFIM [1] masks algorithmic blocks and API calls, Repoformer [2] includes function-body completion in CrossCodeLongEval, and M²RC-EVAL [3] selects completion positions from syntax tree nodes parsed with Tree-sitter.

[1] Gong, L., Wang, S., Elhoushi, M., & Cheung, A. (2024). Evaluation of llms on syntax-aware code fill-in-the-middle tasks. arXiv preprint arXiv:2403.04814.

[2] Wu, D., Ahmad, W. U., Zhang, D., Ramanathan, M. K., & Ma, X. (2024). Repoformer: Selective retrieval for repository-level code completion. arXiv preprint arXiv:2403.10059.

[3] Liu, J., Deng, K., Liu, C., Yang, J., Liu, S., Zhu, H., ... & Zheng, B. (2025, July). M2rc-eval: Massively multilingual repository-level code completion evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 15661-15684).

Dear Reviewer nCmL, thank you again for your constructive feedback. As the discussion period closes on November 18 (AoE), we would greatly appreciate your feedback on our rebuttal and revised manuscript. We added accuracy and latency comparisons with DeepSeek Flash and Codex CLI, provided confidence intervals on RTL-Repo, and clarified training–test overlap screening and the motivation for our masking strategy. We hope everything is clear and the recent experiments have addressed the concerns. Thank you for your time and consideration.

Reviewer_RoNbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

CoRTL addresses repository-level register-transfer level (RTL) code completion in the fill-in-the-middle setting, where the declarations a completion needs often span multiple retrieved chunks. The authors show that selecting chunks by individual utility frequently misses higher-utility combinations. To address this, CoRTL introduces combination utility search, a beam search in which a fixed Qwen2.5-Coder-7B generator scores chunk combinations by how much they reduce per-token cross-entropy on the reference code. The selected combinations supervise a 0.6B reranker and provide context for fine-tuning a 7B completion generator. The authors also construct 47,647 RTL training examples and RealBench-FIM, a benchmark that evaluates functional correctness with testbenches. CoRTL achieves 75.31% func@1 on RealBench-FIM and 66.78% exact match on RTL-Repo. It outperforms RLCoder, RepoShapley, and GPT-5.4 but trails Claude Opus 4.6.

Critical strengths
  1. Reasonable motivation: RTL has 2.44× more cross-file references than Python; top-k selection misses best combinations for 79–98% of targets.
  2. Direct combination scoring: The generator evaluates actual chunk combinations, avoiding surrogates; costly search runs only offline.
  3. Functional benchmark: RealBench-FIM tests completions via testbench simulation in real IP projects, overcoming exact-match limitations.
Critical weaknesses
  1. Unattributed gains: Generator SFT drives most improvement (e.g., 48.31% → 75.31% func@1). However, SFT on non-searched contexts is untested.
  2. Unfair commercial comparisn: Commercial models receive no retrieved context, so claims of surpassing GPT-5.4 are unsupported.
  3. Weak statistical evidence: RealBench-FIM has 145 targets from three designs, without confidence intervals or significance tests.
  4. Unvalidated proxy utility: Cross-entropy may reward lexical copying, and its correlation with functional correctness is never measured.
Questions for authors

Why code completion in this manner in the context of agentic code generation? Isn't this dated in 2026?

3 comments

We thank the reviewer for the detailed feedback and for highlighting the value of scoring context combinations directly and using testbenches to evaluate functional correctness.

W1:

Unattributed gains: Generator SFT drives most improvement (e.g., 48.31% → 75.31% func@1). However, SFT on non-searched contexts is untested.

To test whether CoRTL improves on generator SFT without combination utility search, we added two SFT controls using either only in-file context (no retrieval) or contexts from single-chunk utility selection. On RealBench-FIM, CoRTL achieved 75.31% func@1, versus 67.00% for no retrieval and 74.55% for single-chunk utility. We modified Section 4.4 to include the full-system ablation (Table 3).

Systemfunc@1func@5
No retrieval67.00%69.43%
Single-chunk utility74.55%75.97%
Combination utility75.31%76.36%

W2:

Unfair commercial comparison: Commercial models receive no retrieved context, so claims of surpassing GPT-5.4 are unsupported.

We noticed that using a reranker to select relevant repository context is itself one of CoRTL's contributions, so giving commercial models the context retrieved by CoRTL would make the comparison unfair to CoRTL. Nevertheless, as requested, we evaluated the commercial models with the same retrieved context on the 145 RealBench-FIM tasks. CoRTL remained competitive, achieving 75.31% func@1, just 1.24% below GPT-5.4 (76.55%) but still higher than GLM-5.2 and DeepSeek-V3.2. The improvement on commercial models also shows the effectiveness of our retriever. Appendix A.6 now includes the shared-context experiment (Table 9).

ModelWithout retrieved contextWith CoRTL’s retrieved context
CoRTL-7B-75.31%
GPT-5.468.97%76.55%
GLM-5.261.38%72.41%
DeepSeek-V3.266.90%71.72%

W3:

Weak statistical evidence: RealBench-FIM has 145 targets from three designs, without confidence intervals or significance tests.

We added paired 95% confidence intervals to assess the reliability of CoRTL's gains and found that CoRTL outperformed RLCoder and RepoShapley on both RealBench-FIM and RTL-Repo. We used 20,000 paired bootstrap resamples of the completion targets on each benchmark. The table reports CoRTL minus each baseline in%, with paired 95% confidence intervals in brackets.

  1. On RealBench-FIM (145 targets), CoRTL significantly improves func@1 over RLCoder and RepoShapley. CoRTL also scores higher than GPT-5.4; the paired intervals for its gains over DeepSeek-V3.2 and GLM-5.2 are entirely above zero.
  2. On RTL-Repo (1,174 targets), CoRTL also significantly improves exact match over RLCoder, RepoShapley, GPT-5.4, DeepSeek-V3.2, and GLM-5.2.

Table 10 in Appendix A.8 gives the 95% confidence intervals for each method on both benchmarks.

BaselineRealBench-FIM Δfunc@1 (pp)RTL-Repo ΔEM (pp)
RLCoder+30.83 [23.83, 38.00]+4.69 [2.39, 7.07]
RepoShapley+13.34 [7.48, 19.55]+5.37 [2.81, 8.01]
GPT-5.4+6.34 [−0.76, 13.83]+4.17 [1.36, 6.99]
DeepSeek-V3.2+8.41 [0.62, 16.35]+14.74 [11.75, 17.80]
GLM-5.2+13.93 [5.48, 22.45]+7.07 [4.26, 9.97]

W4:

Unvalidated proxy utility: Cross-entropy may reward lexical copying, and its correlation with functional correctness is never measured.

To answer your question, we have added new experiments on RealBench-FIM (which is the only benchmark that can measure functional correctness), and we show that lower reference-code cross-entropy (CE) is correlated with higher testbench pass rates across the evaluated search states.

  1. We analyzed 582 search states with at least one retrieved chunk across 129 tasks. For each state, we measured CE on the reference code given the chunks selected up to that step, generated one completion, and checked it with the original testbench.
  2. Across the evaluated search states, a randomly selected passing state had a 71.6% chance of having lower reference-code CE than a randomly selected failing state (AUROC 0.716, 95% CI [0.626, 0.801]).

The revised Section 4.4 analyzes the relationship between cross-entropy and functional correctness (Figure 4(a)).

Q1:

Why code completion in this manner in the context of agentic code generation? Isn't this dated in 2026?

Fill-in-the-middle still plays an important role in mordern development when the users want to control the quality of the code and requires a quick interaction, especially in bug sensitive areas like hardware design. We further evaluated Codex CLI with GPT-5.4 on the same 145 RealBench-FIM tasks to compare our results. With Codex CLI, CoRTL came within 3.31% of the Codex CLI setup in func@1 and averaged 1.30 s per task versus 41.42 s, making it well suited to frequent, interactive RTL completion. We added the Codex comparison in Section 4.5 and Figure 4(b); Appendix A.7 describes the evaluation setup.

Setupfunc@1Mean measured time (s/task)
CoRTL75.31%1.30
Codex CLI with GPT-5.478.62%41.42

Dear Reviewer RoNb, thank you again for your constructive feedback. As the discussion period closes on November 18 (AoE), we would greatly appreciate your feedback on our rebuttal and revised manuscript. We added controls for generator fine-tuning, commercial-model comparisons with shared retrieved context, confidence intervals, and an analysis of cross-entropy and functional correctness. We also evaluated Codex CLI to clarify the practical role of FIM. We hope everything is clear and the recent experiments have addressed the concerns. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Revision

Dear reviewers, thank you for your comments and suggestions. We have revised the manuscript in response to your feedback. The main revisions are highlighted in blue in the manuscript. The changes are summarized below:

  1. [Section 4.1, Table 1, and Appendix A.6] We added three specialized FIM baselines on RealBench-FIM and RTL-Repo: StarCoderBase-7B, StarCoder2-15B, and Qwen3-Coder-30B-A3B-Instruct.
  2. [Section 4.4 and Table 3] We added full-system ablations comparing no retrieval, single-chunk utility, and combination utility.
  3. [Section 4.4 and Figure 4(a)] We examined the relationship between cross-entropy on the reference code and functional correctness across 582 search states.
  4. [Section 4.5, Figure 4(b–c), and Appendix A.7] We added completion accuracy and latency comparisons with DeepSeek-V4.1-Flash and Codex CLI with GPT-5.4, together with a breakdown of CoRTL’s inference time.
  5. [Appendix A.6 and Table 9] We evaluated commercial models using the same retrieved context as CoRTL on RealBench-FIM.
  6. [Appendix A.8 and Table 10] We added per-method 95% confidence intervals for func@1 on RealBench-FIM and for exact match and edit similarity on RTL-Repo.
  7. [Appendix A.9 and Tables 11–13] We added experiments with a pretrained Jina reranker and a fixed fine-tuned generator, an alternative generator and reranker, and different reranker training data sizes.
  8. [Appendix A.9 and Table 14] We added a proof-of-concept comparison of combination utility search and single-chunk utility selection on Python RepoEval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.