- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark and a framework for R programming language code search in the biomedical domain. It proposes a 4 staged pipeline for the dataset construction and breaks down the retrieved functions into 3 groups depending on their external and internal dependencies. The evaluation is done under 4 settings: a standard setting which keeps the functions as it is from the source repositories, a neutralized to normalize identifiers name, a randomized, a bytecode and an AST representations to analyse models sensitivity to lexical cues rather than to semantic representation of those functions. Open-source embedding models are used for this evaluation. Results show that models perform better under standard settings, suggesting current models’ reliance on explicit lexical cues.
Critical strengths
-
The authors bring a much needed dataset, framework and benchmark contribution for domain specific text-to-code retrieval in R language. Contributions in that subfield are scarce and effective code retrieval offers valuable industrial application scenarios.
-
The authors provide a fair behavioral comparison between direct lexical cues and semantical equivalents within code snippets retrieved by embedding models. Their study is reinforced by disclosing a transparent methodology (data statistics, prompts, models, settings) which can further help for reproducibility with an interpretable LLM/human alignment study to strengthen the robustness of the designed queries and error analysis.
Critical weaknesses
-
One critical flaw is the class imbalance for group 3 in comparison with group 1 and group 2 functions (l.148-158). I believe that for a benchmark purpose the justification of the selected number of pairs must be provided as well. The authors omitted those details. Likewise, hypothesis testing (l.248) is achieved with a certain LLM, authors should provide the name of the model and additionally discuss the effect of choosing another LLM.
-
Authors should be explicit about the number of retrieved candidates for MAP, MRR and NDCG scores.
Questions for authors
The authors have split repositories to build the training set and the evaluation set (l.169-170). Was it done randomly ? Does the split affect the training and evaluation in any way if different splits are tested ? (It may be negligible, but I would suggest justifying that point for completeness.)
Have you tried to finetune under Bytecode and AST settings ? If so, what conclusions have you observed ?
“by string-typed data accessors, which account for nearly half of the accuracy loss on container-class functions.” (l.83-84) Can you cite the supporting table(s) here ? Based on Table 3,4,5 and 6 recall@1 would be the accuracy here and I can’t see a consistent half accuracy cut.
6 comments
Response to Reviewer doiA - Part 2
Continued after Part 1...
Sensitivity to alternative query generators. To investigate whether the choice of query generator affects the benchmark results, we regenerated queries for the 375 hypothesis test functions using two additional model families: Gemini 2.5 Pro and Qwen3 235B. We retained the same prompts, rescored 50 functions per group with the same annotators, and reran the retrieval experiments.
Table R1. Sensitivity to the query generator (375 functions).
| Check | Result |
|---|---|
| Annotators' verdict (LLM at least as good as expert) | Unchanged in all groups |
| Largest NDCG change, R capable models | 2.5 (CodeT5+: 4.5) |
| Model ranking, Kendall's tau versus original | At least 0.87 (Voyage first, BM25 and CodeT5+ last) |
| Largest change in a Standard to anonymized performance gap | 2.0 |
These results are consistent with the expert written subset reported in Appendix F.2. Replacing LLM generated queries with expert written queries changes Voyage code 3's scores by at most 2.1 points on Group 1 and Group 2 metrics, while Voyage code 3 remains ahead of CodeT5+ in every group.
Thus, the query generators are explicitly identified, the description quality evaluation is human assessed, and the additional experiments show that changing the generator affects NDCG by at most 2.5 points for the R capable models, with the overall conclusions and model rankings remaining stable.
W3. Number of retrieved candidates for MAP, MRR, and NDCG
Response: Thank you for pointing out this missing clarification. No retrieval cutoff is applied when computing MAP, MRR, or NDCG. For each query, every model ranks the entire candidate pool of its respective group, and all three metrics are computed over this full ranking (Appendix E, L1056–1069 and L1075–1088).
The candidate pools contain 493 snippets for Group 1, 462 snippets for Group 2, and 253 snippets for Group 3 Long. For Group 3 Short, the pool contains the 253 primary functions together with their helper functions (L1057). Recall@k is separately reported for k ∈ {1, 5, 10, 20} in Table 3.
Manuscript revision. We will explicitly state the candidate pool sizes and the absence of a retrieval cutoff. This clarification, together with the released evaluation pools, makes the metric computation reproducible.
References
[1] Chen et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.
[2] Husain et al. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv:1909.09436, 2019.
Response to Questions
Q1. How the repository split was made, and whether it matters
Response: Thank you for this question. Our data split is performed at the repository level to prevent repository specific information from leaking between training and evaluation. All functions, helper functions, and near duplicates originating from a repository remain in the same split. Consequently, the 99 training repositories are disjoint from the 45 evaluation repositories (L169–170, Appendix C.1). Repository assignment was performed randomly using a fixed seed.
Additional experiments with five alternative splits. To assess the sensitivity of our findings to the particular repository partition, we generated five additional random repository level splits of the same sizes, stratified by source (Bioconductor versus GitHub). Since every function already has a pipeline generated query, no additional query generation was required. For each split, we evaluated all models in the zero shot setting and fine tuned CodeT5+ and OASIS following the procedure described in Section 5.4.
Table R2. Variation across five repository splits (standard deviation).
| Quantity | Result |
|---|---|
| NDCG of each model, Group 1 / Group 2 / Group 3 | SD at most 1.5 / 1.5 / 3.0 |
| Standard to Neutralized performance gap | SD at most 2.0 |
| CodeT5+ fine tuning gain, Group 1 Neutralized MRR (originally +36.2 in Table 6) | +35 ± 3 |
The observed variation across splits is at most 3.0 NDCG points, substantially smaller than the Group 2 anonymization gaps of 15.3 to 55.6 points (Table 4). These results indicate that the observed performance differences are not sensitive to the particular repository partition.
Manuscript revision. We will include the split procedure and the results from the five alternative partitions in Appendix C.1.
The repository disjoint split prevents repository level leakage, and the additional experiments confirm that both zero shot retrieval performance and fine tuning results remain stable across alternative partitions, with a maximum standard deviation of 3.0 NDCG points.
To be continued...
Response to Reviewer doiA - Part 3
Continued after Part 2...
Q2. Fine tuning on Bytecode and AST
Response: Thank you for this suggestion. In the original submission, we fine tuned the models only on the Standard, Neutralized, and Randomized settings (Table 6). Following your suggestion, we constructed Bytecode and AST representations of the 5,310 training pairs using the same data processing pipeline. We then fine tuned CodeT5+ and OASIS on these representations using the procedure described in Section 5.4 (Appendix F.4).
Table R3. MRR after fine tuning on alternative code representations (Group 1 / Group 2 / Group 3).
| Model | Training setting | Bytecode | AST |
|---|---|---|---|
| CodeT5+ | Without fine tuning (Table 5) | 5.7 / 4.4 / 2.1 | 10.2 / 7.1 / 3.1 |
| Fine tuned on Bytecode | 15 / 10 / 4 | 14 / 9 / 4 | |
| Fine tuned on AST | 8 / 6 / 3 | 36 / 18 / 6 | |
| OASIS | Without fine tuning (Table 5) | 27.5 / 25.1 / 12.4 | 46.9 / 41.6 / 23.0 |
| Fine tuned on Bytecode | 40 / 33 / 16 | 50 / 43 / 24 | |
| Fine tuned on AST | 31 / 27 / 13 | 55 / 47 / 27 |
These experiments reveal four observations.
1. Fine tuning improves performance within the corresponding representation. For example, OASIS fine tuned on Bytecode gains 12.5 MRR points in Group 1, while CodeT5+ fine tuned on AST gains 25.8 points.
2. Fine tuning does not close the representation gap. Despite the improvements, lowered representations remain substantially behind source code. For example, OASIS reaches at most 55 MRR in Group 1 on AST, compared with 83.9 on the Standard setting. This persistent gap is consistent with the anonymization gap reported in Table 6.
3. Transfer between Bytecode and AST is limited. Fine tuning on one lowered representation provides only weak gains when evaluated on the other, with improvements of 2.3 to 3.8 MRR points in Group 1.
4. Model limitations and over specialization remain relevant. CodeT5+'s Bytecode gains are constrained by its 512 token context window, which truncates most Bytecode sequences (Section 5.3). Furthermore, fine tuning OASIS on lowered representations reduces its performance on source code by 3 to 5 MRR points, consistent with the over specialization observed in Table 6.
Overall, the additional experiments show that supervised adaptation improves retrieval within Bytecode and AST representations, but does not eliminate their performance gap with source code. For OASIS in Group 1, the gap remains at least 28.9 MRR points. This extends our original finding that supervised adaptation does not close the representation gap across all five settings.
Q3. Which table supports "nearly half of the accuracy loss"
Response: Thank you for catching this. We agree that the original statement should explicitly identify its supporting table and evaluation metric.
The claim is based on the accessor literal ablation reported in Table 8 (Appendix C.4). Specifically, "half of the loss" refers to the proportion of the Standard to Neutralized performance drop in Group 2 that is recovered by restoring only the accessor literals. The ablation reports NDCG and MRR, which explains why this result is not directly visible in Tables 3 to 6.
Table R4. Proportion of the Group 2 Standard to Neutralized performance loss recovered by intact accessor literals (computed from Tables 3, 4, and 8).
| Model | NDCG | MRR | R@1 |
|---|---|---|---|
| CodeT5+ | 0.41 | 0.39 | 0.40 |
| Nomic | 0.54 | 0.52 | 0.54 |
| OASIS | 0.59 | 0.57 | 0.58 |
| OpenAI | 0.54 | 0.52 | 0.55 |
| Voyage | 0.53 | 0.52 | 0.55 |
Across the five models, restoring accessor literals recovers 41% to 59% of the performance loss in NDCG (mean 52%), 39% to 57% in MRR (mean 50%), and 40% to 58% in R@1 (mean 52%). Four of the five models recover at least 52% on every metric, indicating that the original phrase "nearly half" is a conservative description of the observed effect.
We will explicitly cite Table 8 (Appendix C.4) when discussing this finding and clarify the corresponding metrics. The additional R@1 results also show that the effect holds across ranking quality and top rank retrieval accuracy.
To be continued...
Response to Reviewer doiA - Part 1
We thank Reviewer doiA for recognizing BRACE as a dataset, framework, and benchmark for domain specific text to code retrieval in R. We appreciate the positive assessment of its comparison between lexical cues and semantic equivalents, as well as the reproducibility supported by the reported data statistics, prompts, models, experimental settings, LLM and human alignment study, and error analysis.
The reviewer raised several questions concerning the evaluation group sizes, query generation, repository splitting, and fine tuning on alternative code representations. In response, we conducted four additional analyses: (1) a size matched evaluation of Group 3, (2) a sensitivity analysis using alternative query generators, (3) five independently redrawn repository splits, and (4) additional fine tuning experiments on Bytecode and AST representations. We also clarified the metric computation and provided further evidence supporting the accessor literal ablation. We address each concern below. Paper pointer (old pdf) is in supplimentary and changes is in revised pdf.
Response to Weaknesses
W1. Group 3 size and justification of the number of pairs
Response: Thank you for raising this concern. We agree that the group sizes and their implications for evaluation should be clearly justified.
BRACE is a retrieval benchmark, so group sizes represent evaluation set sizes rather than class frequencies. The reported counts of 493, 462, and 253 are the yields of our data processing pipeline across the 45 evaluation repositories, rather than predetermined sampling targets (L181–182). Group 3 is smaller because a function is included only when its complete intra repository call graph can be extracted and resolved (L175–179).
Evaluation independence across groups. No reported metric pools or combines queries across groups. All metrics are computed separately over each group's own candidate pool (Appendix E). Therefore, differences in group sizes do not introduce cross group weighting effects. The smaller candidate pool in Group 3 also provides fewer distractors, which would generally make retrieval easier. Despite this, Group 3 remains the hardest group under anonymization.
Statistical support for Group 3. All Group 3 conclusions are based on paired, within group comparisons using the same 253 queries. A distribution free Hoeffding bound for paired per query NDCG differences, which lie in [−100, 100], gives a 95% margin of ±17.1 points at n = 253. This margin is smaller than every Standard to Neutralized performance drop observed for the four R capable models in Group 3 Long (26.3 to 49.1 points; Tables 3 and 4) and Group 3 Short (36.0 to 42.8 points). Bootstrap 95% confidence intervals are tighter, with a margin of approximately ±4 points for Voyage code 3's Group 3 Long performance drop.
Additional size matched analysis. Group 3 is also the most extensively validated and code intensive group. Our hypothesis test involving 125 functions covers 49% of Group 3, compared with 25% of Group 1 and 27% of Group 2. Furthermore, at 612.3 tokens per item (Table 1), Group 3 accounts for 59.7% of all evaluated source tokens.
To examine whether the smaller evaluation size itself explains the observed difficulty, we subsampled Groups 1 and 2 to 253 pairs and repeated the evaluation over 1,000 draws. The results show that subsampling raises their retrieval scores and reduces their anonymization gaps. For example, the Standard to Neutralized NDCG gap for Voyage code 3 in Group 2 decreases from 15.3 to 11 points. Thus, the difficulty observed in Group 3 cannot be attributed simply to its smaller evaluation size.
The evaluation sizes are also comparable to established code evaluation benchmarks, including HumanEval with 164 problems [1] and the CodeSearchNet Challenge with 99 expert annotated queries [2].
Manuscript revision. We will clarify that the group sizes are the pipeline yields across the 45 evaluation repositories: 493, 462, and 253 pairs for Groups 1, 2, and 3, respectively. We will also explicitly state that all metrics are computed independently over each group's candidate pool.
Overall, the dependency resolution requirement explains the smaller Group 3 size, while the size matched evaluation and paired statistical analysis show that its observed anonymization effects are not artifacts of the smaller sample.
W2. Name of the LLM and the effect of choosing another LLM
Response: Thank you for this observation. Section 3.1.4 already identifies both query generators, o3-mini and grok-4 (L239), and their prompts are provided in Appendix K. We also clarify that the hypothesis test is conducted by human annotators who compare LLM generated descriptions with expert descriptions without knowing their sources (L250–260). Therefore, this evaluation measures description quality through human assessment rather than relying on an LLM judge.
To be continued...