acceptodds
Under review as a conference paper at ICLR 2027

BRACE: An R Benchmark for Rigorous Biomedical Data Science Code Search

Abstract

Efficient code retrieval is critical for biomedical data scientists, who navigate thousands of R packages on CRAN and Bioconductor and a growing body of open-source analysis pipelines. Existing code search benchmarks, however, focus on Python and rarely stress-test robustness beyond superficial lexical cues. To address this gap, we adapt an automated benchmark-construction pipeline to R and present BRACE (Biomedical R Anonymized Code rEtrieval), a benchmark built from real-world biomedical repositories and packages. BRACE contains 1,208 query-code pairs for evaluation and 5,310 pairs for training. Queries are LLM-generated natural-language descriptions validated through scoring by domain experts and hypothesis testing. The pipeline first ensures that every snippet byte-compiles and resolves all of its dependencies inside a pinned R/Bioconductor environment. It then categorizes snippets by dependency complexity, distinguishing functions that use only base R, functions that rely on custom S3/S4/R6 or Bioconductor container classes, and functions that invoke user-defined helpers. BRACE further stress-tests retrieval robustness through identifier anonymization and through representation shift, in which functions are lowered to R bytecode or serialized as S-expression parse trees. Under these conditions, our evaluation of six retrieval models reveals consistent drops when identifiers are anonymized, NDCG falls by 3-25 points on base-R functions and by up to 56 points on container-class functions, and far larger drops on lowered representations, where even the strongest model ranks the correct function first for fewer than a quarter of bytecode queries. The results indicate that current models still rely on lexical features rather than code semantics, and that in R this reliance extends to string-typed data accessors.

Primary area:
datasets and benchmarks
Keywords:
Code search, Code retrieval, Benchmark, R, Biomedical Data Science, Retrieval-Augmented Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_doiAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark and a framework for R programming language code search in the biomedical domain. It proposes a 4 staged pipeline for the dataset construction and breaks down the retrieved functions into 3 groups depending on their external and internal dependencies. The evaluation is done under 4 settings: a standard setting which keeps the functions as it is from the source repositories, a neutralized to normalize identifiers name, a randomized, a bytecode and an AST representations to analyse models sensitivity to lexical cues rather than to semantic representation of those functions. Open-source embedding models are used for this evaluation. Results show that models perform better under standard settings, suggesting current models’ reliance on explicit lexical cues.

Critical strengths
  • The authors bring a much needed dataset, framework and benchmark contribution for domain specific text-to-code retrieval in R language. Contributions in that subfield are scarce and effective code retrieval offers valuable industrial application scenarios.

  • The authors provide a fair behavioral comparison between direct lexical cues and semantical equivalents within code snippets retrieved by embedding models. Their study is reinforced by disclosing a transparent methodology (data statistics, prompts, models, settings) which can further help for reproducibility with an interpretable LLM/human alignment study to strengthen the robustness of the designed queries and error analysis.

Critical weaknesses
  • One critical flaw is the class imbalance for group 3 in comparison with group 1 and group 2 functions (l.148-158). I believe that for a benchmark purpose the justification of the selected number of pairs must be provided as well. The authors omitted those details. Likewise, hypothesis testing (l.248) is achieved with a certain LLM, authors should provide the name of the model and additionally discuss the effect of choosing another LLM.

  • Authors should be explicit about the number of retrieved candidates for MAP, MRR and NDCG scores.

Questions for authors

The authors have split repositories to build the training set and the evaluation set (l.169-170). Was it done randomly ? Does the split affect the training and evaluation in any way if different splits are tested ? (It may be negligible, but I would suggest justifying that point for completeness.)

Have you tried to finetune under Bytecode and AST settings ? If so, what conclusions have you observed ?

“by string-typed data accessors, which account for nearly half of the accuracy loss on container-class functions.” (l.83-84) Can you cite the supporting table(s) here ? Based on Table 3,4,5 and 6 recall@1 would be the accuracy here and I can’t see a consistent half accuracy cut.

6 comments
AuthorsOpenReview ↗

Response to Reviewer doiA - Part 1

We thank Reviewer doiA for recognizing BRACE as a dataset, framework, and benchmark for domain specific text to code retrieval in R. We appreciate the positive assessment of its comparison between lexical cues and semantic equivalents, as well as the reproducibility supported by the reported data statistics, prompts, models, experimental settings, LLM and human alignment study, and error analysis.

The reviewer raised several questions concerning the evaluation group sizes, query generation, repository splitting, and fine tuning on alternative code representations. In response, we conducted four additional analyses: (1) a size matched evaluation of Group 3, (2) a sensitivity analysis using alternative query generators, (3) five independently redrawn repository splits, and (4) additional fine tuning experiments on Bytecode and AST representations. We also clarified the metric computation and provided further evidence supporting the accessor literal ablation. We address each concern below. Paper pointer (old pdf) is in supplimentary and changes is in revised pdf.


Response to Weaknesses

W1. Group 3 size and justification of the number of pairs

Response: Thank you for raising this concern. We agree that the group sizes and their implications for evaluation should be clearly justified.

BRACE is a retrieval benchmark, so group sizes represent evaluation set sizes rather than class frequencies. The reported counts of 493, 462, and 253 are the yields of our data processing pipeline across the 45 evaluation repositories, rather than predetermined sampling targets (L181–182). Group 3 is smaller because a function is included only when its complete intra repository call graph can be extracted and resolved (L175–179).

Evaluation independence across groups. No reported metric pools or combines queries across groups. All metrics are computed separately over each group's own candidate pool (Appendix E). Therefore, differences in group sizes do not introduce cross group weighting effects. The smaller candidate pool in Group 3 also provides fewer distractors, which would generally make retrieval easier. Despite this, Group 3 remains the hardest group under anonymization.

Statistical support for Group 3. All Group 3 conclusions are based on paired, within group comparisons using the same 253 queries. A distribution free Hoeffding bound for paired per query NDCG differences, which lie in [−100, 100], gives a 95% margin of ±17.1 points at n = 253. This margin is smaller than every Standard to Neutralized performance drop observed for the four R capable models in Group 3 Long (26.3 to 49.1 points; Tables 3 and 4) and Group 3 Short (36.0 to 42.8 points). Bootstrap 95% confidence intervals are tighter, with a margin of approximately ±4 points for Voyage code 3's Group 3 Long performance drop.

Additional size matched analysis. Group 3 is also the most extensively validated and code intensive group. Our hypothesis test involving 125 functions covers 49% of Group 3, compared with 25% of Group 1 and 27% of Group 2. Furthermore, at 612.3 tokens per item (Table 1), Group 3 accounts for 59.7% of all evaluated source tokens.

To examine whether the smaller evaluation size itself explains the observed difficulty, we subsampled Groups 1 and 2 to 253 pairs and repeated the evaluation over 1,000 draws. The results show that subsampling raises their retrieval scores and reduces their anonymization gaps. For example, the Standard to Neutralized NDCG gap for Voyage code 3 in Group 2 decreases from 15.3 to 11 points. Thus, the difficulty observed in Group 3 cannot be attributed simply to its smaller evaluation size.

The evaluation sizes are also comparable to established code evaluation benchmarks, including HumanEval with 164 problems [1] and the CodeSearchNet Challenge with 99 expert annotated queries [2].

Manuscript revision. We will clarify that the group sizes are the pipeline yields across the 45 evaluation repositories: 493, 462, and 253 pairs for Groups 1, 2, and 3, respectively. We will also explicitly state that all metrics are computed independently over each group's candidate pool.

Overall, the dependency resolution requirement explains the smaller Group 3 size, while the size matched evaluation and paired statistical analysis show that its observed anonymization effects are not artifacts of the smaller sample.

W2. Name of the LLM and the effect of choosing another LLM

Response: Thank you for this observation. Section 3.1.4 already identifies both query generators, o3-mini and grok-4 (L239), and their prompts are provided in Appendix K. We also clarify that the hypothesis test is conducted by human annotators who compare LLM generated descriptions with expert descriptions without knowing their sources (L250–260). Therefore, this evaluation measures description quality through human assessment rather than relying on an LLM judge.


To be continued...

AuthorsOpenReview ↗

Response to Reviewer doiA - Part 2

Continued after Part 1...


Sensitivity to alternative query generators. To investigate whether the choice of query generator affects the benchmark results, we regenerated queries for the 375 hypothesis test functions using two additional model families: Gemini 2.5 Pro and Qwen3 235B. We retained the same prompts, rescored 50 functions per group with the same annotators, and reran the retrieval experiments.

Table R1. Sensitivity to the query generator (375 functions).

CheckResult
Annotators' verdict (LLM at least as good as expert)Unchanged in all groups
Largest NDCG change, R capable models2.5 (CodeT5+: 4.5)
Model ranking, Kendall's tau versus originalAt least 0.87 (Voyage first, BM25 and CodeT5+ last)
Largest change in a Standard to anonymized performance gap2.0

These results are consistent with the expert written subset reported in Appendix F.2. Replacing LLM generated queries with expert written queries changes Voyage code 3's scores by at most 2.1 points on Group 1 and Group 2 metrics, while Voyage code 3 remains ahead of CodeT5+ in every group.

Thus, the query generators are explicitly identified, the description quality evaluation is human assessed, and the additional experiments show that changing the generator affects NDCG by at most 2.5 points for the R capable models, with the overall conclusions and model rankings remaining stable.

W3. Number of retrieved candidates for MAP, MRR, and NDCG

Response: Thank you for pointing out this missing clarification. No retrieval cutoff is applied when computing MAP, MRR, or NDCG. For each query, every model ranks the entire candidate pool of its respective group, and all three metrics are computed over this full ranking (Appendix E, L1056–1069 and L1075–1088).

The candidate pools contain 493 snippets for Group 1, 462 snippets for Group 2, and 253 snippets for Group 3 Long. For Group 3 Short, the pool contains the 253 primary functions together with their helper functions (L1057). Recall@k is separately reported for k ∈ {1, 5, 10, 20} in Table 3.

Manuscript revision. We will explicitly state the candidate pool sizes and the absence of a retrieval cutoff. This clarification, together with the released evaluation pools, makes the metric computation reproducible.

References

[1] Chen et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.

[2] Husain et al. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv:1909.09436, 2019.


Response to Questions

Q1. How the repository split was made, and whether it matters

Response: Thank you for this question. Our data split is performed at the repository level to prevent repository specific information from leaking between training and evaluation. All functions, helper functions, and near duplicates originating from a repository remain in the same split. Consequently, the 99 training repositories are disjoint from the 45 evaluation repositories (L169–170, Appendix C.1). Repository assignment was performed randomly using a fixed seed.

Additional experiments with five alternative splits. To assess the sensitivity of our findings to the particular repository partition, we generated five additional random repository level splits of the same sizes, stratified by source (Bioconductor versus GitHub). Since every function already has a pipeline generated query, no additional query generation was required. For each split, we evaluated all models in the zero shot setting and fine tuned CodeT5+ and OASIS following the procedure described in Section 5.4.

Table R2. Variation across five repository splits (standard deviation).

QuantityResult
NDCG of each model, Group 1 / Group 2 / Group 3SD at most 1.5 / 1.5 / 3.0
Standard to Neutralized performance gapSD at most 2.0
CodeT5+ fine tuning gain, Group 1 Neutralized MRR (originally +36.2 in Table 6)+35 ± 3

The observed variation across splits is at most 3.0 NDCG points, substantially smaller than the Group 2 anonymization gaps of 15.3 to 55.6 points (Table 4). These results indicate that the observed performance differences are not sensitive to the particular repository partition.

Manuscript revision. We will include the split procedure and the results from the five alternative partitions in Appendix C.1.

The repository disjoint split prevents repository level leakage, and the additional experiments confirm that both zero shot retrieval performance and fine tuning results remain stable across alternative partitions, with a maximum standard deviation of 3.0 NDCG points.


To be continued...

AuthorsOpenReview ↗

Response to Reviewer doiA - Part 3

Continued after Part 2...


Q2. Fine tuning on Bytecode and AST

Response: Thank you for this suggestion. In the original submission, we fine tuned the models only on the Standard, Neutralized, and Randomized settings (Table 6). Following your suggestion, we constructed Bytecode and AST representations of the 5,310 training pairs using the same data processing pipeline. We then fine tuned CodeT5+ and OASIS on these representations using the procedure described in Section 5.4 (Appendix F.4).

Table R3. MRR after fine tuning on alternative code representations (Group 1 / Group 2 / Group 3).

ModelTraining settingBytecodeAST
CodeT5+Without fine tuning (Table 5)5.7 / 4.4 / 2.110.2 / 7.1 / 3.1
Fine tuned on Bytecode15 / 10 / 414 / 9 / 4
Fine tuned on AST8 / 6 / 336 / 18 / 6
OASISWithout fine tuning (Table 5)27.5 / 25.1 / 12.446.9 / 41.6 / 23.0
Fine tuned on Bytecode40 / 33 / 1650 / 43 / 24
Fine tuned on AST31 / 27 / 1355 / 47 / 27

These experiments reveal four observations.

1. Fine tuning improves performance within the corresponding representation. For example, OASIS fine tuned on Bytecode gains 12.5 MRR points in Group 1, while CodeT5+ fine tuned on AST gains 25.8 points.

2. Fine tuning does not close the representation gap. Despite the improvements, lowered representations remain substantially behind source code. For example, OASIS reaches at most 55 MRR in Group 1 on AST, compared with 83.9 on the Standard setting. This persistent gap is consistent with the anonymization gap reported in Table 6.

3. Transfer between Bytecode and AST is limited. Fine tuning on one lowered representation provides only weak gains when evaluated on the other, with improvements of 2.3 to 3.8 MRR points in Group 1.

4. Model limitations and over specialization remain relevant. CodeT5+'s Bytecode gains are constrained by its 512 token context window, which truncates most Bytecode sequences (Section 5.3). Furthermore, fine tuning OASIS on lowered representations reduces its performance on source code by 3 to 5 MRR points, consistent with the over specialization observed in Table 6.

Overall, the additional experiments show that supervised adaptation improves retrieval within Bytecode and AST representations, but does not eliminate their performance gap with source code. For OASIS in Group 1, the gap remains at least 28.9 MRR points. This extends our original finding that supervised adaptation does not close the representation gap across all five settings.

Q3. Which table supports "nearly half of the accuracy loss"

Response: Thank you for catching this. We agree that the original statement should explicitly identify its supporting table and evaluation metric.

The claim is based on the accessor literal ablation reported in Table 8 (Appendix C.4). Specifically, "half of the loss" refers to the proportion of the Standard to Neutralized performance drop in Group 2 that is recovered by restoring only the accessor literals. The ablation reports NDCG and MRR, which explains why this result is not directly visible in Tables 3 to 6.

Table R4. Proportion of the Group 2 Standard to Neutralized performance loss recovered by intact accessor literals (computed from Tables 3, 4, and 8).

ModelNDCGMRRR@1
CodeT5+0.410.390.40
Nomic0.540.520.54
OASIS0.590.570.58
OpenAI0.540.520.55
Voyage0.530.520.55

Across the five models, restoring accessor literals recovers 41% to 59% of the performance loss in NDCG (mean 52%), 39% to 57% in MRR (mean 50%), and 40% to 58% in R@1 (mean 52%). Four of the five models recover at least 52% on every metric, indicating that the original phrase "nearly half" is a conservative description of the observed effect.

We will explicitly cite Table 8 (Appendix C.4) when discussing this finding and clarify the corresponding metrics. The additional R@1 results also show that the effect holds across ranking quality and top rank retrieval accuracy.


To be continued...

Reviewer_SQQrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BRACE, a benchmark designed to evaluate the effectiveness and robustness of code retrieval models for biomedical R programming. BRACE consists of 6,518 natural language–code pairs collected from real-world biomedical repositories, with executable code and LLM-generated queries validated by domain experts. Experiments on six retrieval methods reveal substantial performance degradation when identifiers are anonymized or code is transformed into lower-level representations. Further fine-tuning experiments demonstrate that conventional supervised adaptation does not fully address these robustness issues.

Critical strengths

The paper proposes a new dataset.

Critical weaknesses

Overall, the novelty and contribution of this paper are limited. In terms of novelty, the discussions in this paper are already well-known in code retrieval. For example, the authors use identifier anonymization, randomization, bytecode, and abstract syntax tree (AST) representations to assess whether retrieval models capture code semantics rather than superficial lexical patterns. It is widely believed that current retrievers rely more on lexical similarity. In terms of contribution, the size of the proposed dataset is not large enough to support large scale training. And experimental results also demonstrate that current retrievers are doing well on the proposed benchmark, according to Table 3.

Questions for authors

NA

9 comments
AuthorsOpenReview ↗

Response to Reviewer SQQr - Part 1

We thank Reviewer SQQr for recognizing BRACE as a new dataset and for rating its presentation as clear. The reviewer raises three concerns regarding novelty, dataset size, and the results in Table 3. We address these concerns through a clearer distinction between established expectations and new empirical findings, comparisons with existing benchmarks, eight specific results that emerge from BRACE, and an additional experiment using a training set expanded to 10,724 pairs.

Paper pointer (old pdf) is in supplimentary and changes is in revised pdf.


Response to Weaknesses

W1. Novelty: the discussions are already well known in code retrieval

Response: Thank you for this comment. We respectfully disagree with the assessment that the contribution is already established. This assessment appears to conflate three distinct questions: (a) whether code models rely on identifier names, (b) how strongly that reliance affects retrieval across different languages, domains, and code representations, and (c) why the effects occur and whether supervised training can overcome them.

We agree that lexical reliance is an established expectation, and we explicitly acknowledge this in our introduction. Previous studies have demonstrated identifier sensitivity in models of other programming languages [10, 11], while CodeXGLUE, AdvTest evaluates a single name normalization setting for Python code search [2]. However, these findings do not establish how retrieval models behave across different dependency structures, anonymization settings, and code representations in R. BRACE addresses these previously unmeasured questions.

1. BRACE provides the first measurement of these effects in R.

To the best of our knowledge, no previous text to code retrieval benchmark covers R (Sections 1 and 2). This is a meaningful gap because the training corpora of the evaluated models rarely include R: CoRNStack covers six languages without R, and CodeT5+ pretraining covers nine languages (Section 2). Whether strong retrieval performance in R reflects semantic understanding or lexical matching therefore remains an open empirical question in a language central to biomedical data science. Bioconductor alone distributes more than two thousand software packages.

Measuring a suspected model weakness in a previously unstudied language is itself a recognized benchmark contribution. HANS [12], ImageNet C [13], and CheckList [14] measured suspected model fragilities, while AdvTest [2], ReCode [15], and CLARC [5] investigated similar issues in code models for other languages. None of these studies measures retrieval behavior across R code, R specific accessors, and R bytecode.

Table R1 compares BRACE with the most closely related retrieval benchmarks.

Table R1. Coverage of related text to code retrieval benchmarks (to our knowledge, based on the cited papers).

BenchmarkLanguageDependencies resolvedDependency strataName anonymizationLowered form
CodeSearchNet [1]Go, Java, JavaScript, PHP, Python, RubyNoNoNoNo
CodeXGLUE AdvTest [2]PythonNoNoOne setting (func, arg_i)No
CoSQA [3]PythonNoNoNoNo
CoIR [4]Mostly PythonNoNoNoNo
CLARC [5]C/C++Yes (compilable)YesNeutralized, RandomizedAssembly, WebAssembly
BRACER (biomedical)Yes (pinned renv environment)Yes (3 groups, 1 group 2 variants)Neutralized, Randomized, Accessor ablationR bytecode, AST

2. Existing literature does not establish a universal account of lexical reliance.

Probing studies show that code models encode syntactic structure [16, 17], while transformation studies on earlier models report substantial fragility to identifier renaming [10, 11, 18]. However, the behavior of recent embedding models, including OASIS (robustness oriented training) and Nomic emb code (hard negative mining), had not, to our knowledge, been evaluated on R under anonymization.

Our results show that lexical reliance is conditional on model and function type. On base R functions, code specialized models are relatively robust to Neutralized anonymization, with NDCG drops of 3.5 for Voyage, 3.3 for OASIS, and 3.3 for Nomic. On container class functions, the same models experience drops 4.4 to 6.9 times larger: 15.3 for Voyage, 19.2 for OASIS, and 22.9 for Nomic. OpenAI text embedding 3 large loses 44.7 points, while Randomized anonymization produces drops of up to 55.6 points.

Thus, a general expectation that retrievers rely on lexical similarity does not explain the substantial differences across models, function types, and transformations that BRACE measures.


To be continued...

AuthorsOpenReview ↗

Response to Reviewer SQQr - Part 2

Continued after Part 1...


3. BRACE establishes eight empirical findings that are not implied by the general expectation.

A general statement about lexical reliance cannot predict the following results, which emerge from our controlled experiments.

Table R2. Eight empirical findings established by BRACE.

#FindingEvidence
aThe effect of anonymization depends on model and function typeNDCG drops range from 3.3 points (Nomic and OASIS, Group 1 Neutralized) to 55.6 points (OpenAI, Group 2 Randomized) (Table 4).
bStandard accuracy conceals substantial robustness differencesThe NDCG spread among the four R capable models increases from 7.8 points in Group 2 Standard to 41.7 in Randomized (5.3 times larger), and from 5.2 to 19.2 in Group 1 (3.7 times larger). Nomic and OASIS swap positions in Group 2, while OpenAI drops from second (81.2) to fourth (32.1) in Group 3 Long Neutralized.
cContainer class names carry retrieval signalsGroup 2 scores exceed Group 1 by 1.9 to 4.9 NDCG points in Standard for the four R capable models, whereas Group 1 exceeds Group 2 by 7.3 to 31.0 points in Neutralized.
dString typed data accessors constitute a separate retrieval signal in RIntact accessors recover 41% to 59% (mean 52%) of Group 2 Neutralized NDCG loss (Table 8). Accessor dependence accounts for 27% of sampled Neutralized failures, 24% of Randomized failures, and only 2% of bytecode failures (Appendix G).
eHelper context becomes more important when names are removedVoyage's Group 3 Long minus Short NDCG gap increases from 22.7 points in Standard to 34.9 in Neutralized.
fLowered representations fail through a different mechanismThe best bytecode R@1 is 23.5% (Group 1). Signal dilution accounts for 66% of sampled bytecode failures, with sequences averaging 688 tokens in Group 1, compared with only 2% attributable to accessor dependence (Appendix G).
gThe cost of missing language coverage can be quantifiedCodeT5+, whose pretraining excludes R, trails Nomic, OASIS, and Voyage by 38.1 to 39.8 NDCG points in Group 1 Standard.
hSupervised adaptation and additional training data do not eliminate the robustness gapThe Standard to Randomized MRR gap remains positive in all 18 fine tuned configurations (5.4 to 43.2 points, Table 6). Doubling the training set to 10,724 pairs changes the best case gap by −0.7 (Group 1), 0.0 (Group 2), and +1.2 (Group 3) MRR points (Table R4 below).

Results 3, 5, and 6 correspond to our preregistered hypotheses H2, H3, and H4 (Section 5), respectively, meaning that these hypotheses could have been falsified by the experiments. Result 6 also demonstrates that bytecode evaluation captures a failure mechanism beyond lexical reliance: only 2% of sampled bytecode failures involve accessor literals, whereas 66% result from signal dilution.

4. Relevance to biomedical practice.

The transformations evaluated in BRACE reflect practical conditions encountered in scientific software development.

First, installed R packages are byte compiled by default, meaning that retrieval systems indexing installed libraries may operate directly on bytecode (Section 3.1.3). Second, laboratory pipelines frequently use generic variable names such as tmp2 and df_final_v3 (Section 1). Third, equivalent statistical quantities often appear under different column names across tools, such as padj, FDR, and adj.P.Val. A retriever that relies heavily on a particular identifier may therefore miss functionally equivalent implementations (Appendix H).

These issues directly affect retrieval augmented scientific coding agents, which depend on retrieving relevant code to support their generated analyses (Section 1). A retriever that fails under these conditions may return implementations that match on names but differ in behavior. Such performance degradation under semantics preserving transformations is a characteristic of shortcut learning [19]. BRACE measures this behavior specifically for R.


To be continued...

AuthorsOpenReview ↗

Response to Reviewer SQQr - Part 3

Continued after Part 2...


5. Differences from CLARC, the closest prior work.

BRACE adopts the construction and validation protocol of CLARC [5], as acknowledged in Section 2, allowing comparisons across programming languages. However, BRACE introduces several R specific contributions:

a) R native code lowering: We construct bytecode using compiler::cmpfun, retaining its constant pool and representing the form found in installed R packages. We also introduce an S expression AST setting. CLARC instead uses Assembly and WebAssembly generated by external compilers and does not include an AST setting.

b) R specific dependency structures: We define dependency groups using R class systems, including S3, S4, R6, and Bioconductor containers, together with helper functions. This analysis reveals the Group 1 and Group 2 performance reversal reported in Result 3.

c) Accessor literal analysis: We introduce a separate accessor based evaluation axis and its corresponding ablation, establishing the contribution of string typed accessors to retrieval performance (Result 4).

d) Dependency resolution for a dynamically typed language: We implement dependency analysis using codetools and a pinned renv environment based on Bioconductor 3.22. We also provide behavioral validation of transformed code, as described in our response to Reviewer WU8b.

e) Dataset and expert validation: BRACE contains 245 repositories, comprising 58 Bioconductor packages and 187 GitHub repositories. Its expert queries are written by bioinformaticians with at least five years of R experience. We also provide an error taxonomy covering 400 cases across seven categories, with Cohen's kappa of 0.71 (Appendix G).

6. Precedent for benchmark contributions.

CLARC [5] was accepted at ICLR 2026 with a similar contribution for C/C++, comprising 1,245 evaluation pairs and 5,472 training pairs. CoIR [4] was accepted to ACL 2025 Main as a code retrieval benchmark. ScienceAgentBench [6], which includes bioinformatics tasks, was accepted at ICLR 2025 with 102 tasks. MultiPL E [7] similarly extended code generation evaluation across programming languages.

These works establish precedent for extending evaluation methodologies to new languages and domains. BRACE follows this established direction while contributing new domain specific findings, as summarized in Table R2.

Manuscript revision. We will clarify the novelty statement in Section 1 as follows:

"Prior work indicates that code models depend on identifier names [10, 11], while probing studies find that they also encode syntactic structure [16, 17]. BRACE shows that, in R, this dependence is conditional on function type and model architecture: on base R functions, code specialized retrievers lose 3.3 to 3.5 NDCG points under identifier neutralization, whereas on container class functions they lose 15.3 to 22.9 points. Restoring intact string typed accessor literals recovers 41% to 59% of the Group 2 Neutralized NDCG loss across the five embedding models."

The central distinction is that the general expectation of lexical reliance is already known, but its extent, conditions, and underlying mechanisms in R have not been established. BRACE provides the first R text to code retrieval benchmark we are aware of, with resolved dependencies, and reports eight empirical findings that go beyond this expectation (Table R2). Table R1 further demonstrates that no previous benchmark combines R, resolved dependencies, dependency strata, anonymized code, and lowered representations.

W2. Dataset size is not large enough for large scale training

Response: Thank you for this point. BRACE is primarily an evaluation benchmark. Its 1,208 evaluation pairs constitute the benchmark's evaluation resource, while the 5,310 training pairs support experiments investigating whether supervised adaptation can close the observed retrieval gaps (Section 5.4). Nevertheless, we agree that examining the scalability of the data construction pipeline is useful, and we conducted an additional experiment with a substantially larger training set.


To be continued...

Reviewer_WU8bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propposes BRACE, a biomedical R retrieval dataset consisting of 6,518 query-code pairs. It directly tackles the gap that exisitng retrieval benchmarks have limited focus on R, are often not executable, and may not evaluate robustness under textual perturbation. Evaluation on six retrieval models show that though they may perform well on natural source, their performance consistently decline when identifiers are anonymized. It is also observed that retrieval is significant worse on lower-level representations such as bytecode.

Critical strengths
  • The paper tackles Biomedical R retrieval, which is a relevant but underexplored area for code retrieval. The dataset fills up the gap, with 6k query-sample pairs that are compilable and executable.
  • The Neutralized setting is insightful to show that existing retrieval methods rely on lexical features, and the ablation on accessor strings demonstrate that the data-accessor strings carry substantial retrieval information.
Critical weaknesses
  • The anonymization section needs to validate that the removal or masking does not strip away necessary semantic information. For example, in biomedical R, accessor names may carry statistical meaning of a field, and masking these names may leave out important information for retrieval.
  • Correctness validation: the paper does not ensure equivalence of the transformed code. As the authors claim that BRACE is fully executable, there should be some behavioral validation such as tests that ensure the equivalence of the transformed function after masking or renaming.
Questions for authors
  • Will removal of string literals preserve the same amount of semantic information for retrieval?
  • Aside from parsing and byte compilation, are functions in BRACE executed on real, vlaid inputs?
  • Can you compare the original BRACE versus anonymized bytecode/ast setting on a common subset?
7 comments
AuthorsOpenReview ↗

Response to Reviewer WU8b - Part 1

We thank Reviewer WU8b for recognizing that BRACE addresses biomedical R, a domain without an existing text to code retrieval benchmark, and that its 6,518 pairs are compilable. We also appreciate the positive assessment of the Neutralized setting and accessor string ablation, which help identify the lexical features on which retrievers depend.

The reviewer raises three important concerns: whether masking removes semantic information needed for retrieval, whether transformed code preserves program behavior, and whether the retrieval performance differences between source code, AST, and Bytecode can be isolated from anonymization effects. These concerns motivated three additional analyses: (1) a quantitative breakdown of the information removed by masking, (2) execution based behavioral validation of transformed functions, and (3) a side by side comparison of all five code representations using the same evaluation functions. We address each concern below. Paper pointer (old pdf) is in supplimentary and changes is in revised pdf.


Response to Weaknesses and Questions

W1 and Q1. Does masking remove semantic information that retrieval needs? Does removing string literals preserve the same amount of semantic information?

Response: Thank you for raising this concern. We agree that accessor names such as padj can carry statistical meaning in biomedical R. This is precisely why BRACE treats accessor literals as a separate, measurable factor rather than leaving their effects as an unexamined consequence of anonymization.

1. What is masked and what is retained.

BRACE does not delete string literals. Instead, data accessor literals, including $, [[ with a string, @, and tidy evaluation column references, are renamed using the same consistent mapping applied to identifiers. Thus, every occurrence of a field maps to the same placeholder (L203–205, L982–983). Numeric constants, other string literals, and names exported by whitelisted packages, such as p.adjust, colSums, and assay, are retained (L202–203).

The query generator is instructed to describe the underlying computation without using identifier, package, or column names (L245–247). Consequently, the query does not contain a field name that is removed from the corresponding code during anonymization.

2. The contribution of accessor literals is measured through a controlled ablation.

Rather than assuming that accessor masking has no effect on semantic information, we quantify its contribution by restoring only the accessor literals in the Neutralized setting. All other anonymization operations remain unchanged.

Table R1. Group 2 NDCG under different accessor masking conditions (from Tables 3, 4, and 8).

ModelStandardNeutralized, accessors maskedNeutralized, accessors intactLoss recovered by intact accessorsRemaining loss
CodeT5+39.614.824.910.1 (41%)14.7
Nomic90.867.980.212.3 (54%)10.6
OASIS89.570.381.611.3 (59%)7.9
OpenAI84.339.663.724.1 (54%)20.6
Voyage92.176.884.98.1 (53%)7.2

Restoring accessor literals recovers 41% to 59% of the Group 2 NDCG loss, with a mean recovery of 52%. However, identifier anonymization still causes a loss of 7.2 to 20.6 NDCG points when accessor literals remain intact. Therefore, the observed dependence on names cannot be attributed solely to the removal of field names.

The Group 1 results provide a complementary control. Since base R functions rarely access fields through string typed accessors, Neutralized anonymization produces only 3.3 points of NDCG loss for Nomic and OASIS and 3.5 points for Voyage (Tables 3 and 4). This suggests that the anonymization procedure does not indiscriminately remove information required for retrieval when the accessor channel is absent.

3. The masked functions remain identifiable through retained computational structure.

The same masked Group 2 inputs produce substantially different results across models. Voyage retains an NDCG of 76.8 and R@5 of 85.0, whereas OpenAI falls to 39.6 NDCG (Table 4). Furthermore, fine tuning CodeT5+ on Neutralized data raises its Group 2 Neutralized MRR from 11.7 to 33.8 (Table 6).

These results indicate that meaningful retrieval signals remain after anonymization. If masking had removed the information necessary to identify the functions, we would expect a more uniform collapse across models and little opportunity for supervised adaptation.

We also remove near duplicates with identical identifier normalized parse trees during dataset construction (Section 3.1.1). Thus, candidates cannot be distinguished solely through their original names. The failure analysis in Appendix G, Figure 2, further shows that 27% of sampled Neutralized failures involve accessor cases, while 31% are skeleton collisions in which the distinguishing operation remains present in the code.


To be continued...

AuthorsOpenReview ↗

Response to Reviewer WU8b - Part 2

Continued after Part 1...


4. Why accessor literals are masked by default.

R allows the same field to be accessed through multiple syntactic forms, such as @padj, $padj, and [["padj"]]. Slot names accessed through @ are already treated as identifiers in the Neutralized setting. Masking only some accessor forms would therefore leave the same field name visible through other forms of access.

For this reason, we apply a consistent mapping across accessor forms rather than selectively retaining particular representations. The ablation in Table 8 provides the corresponding experiment in which accessor literals are left intact.

Manuscript revision. We will clarify Section 3.1.3 as follows:

"String literals are not deleted during anonymization. Literals that act as data accessors, such as x$padj, x[["counts"]], x@slot, and tidy evaluation column references, are renamed using the same consistent mapping as identifiers. R exposes the same field through multiple accessor forms, and masking only a subset would leave its name visible through the others. Table 8 reports an ablation in which accessor literals are retained, showing that they account for 41% to 59% of the Group 2 Neutralized performance loss."

The controlled ablation separates the contribution of accessor masking from the remaining identifier masking effects. The Group 1 results and fine tuning experiments further demonstrate that substantial computational information remains available after anonymization.

W2 and Q2. Correctness validation of transformed code and execution on real valid inputs

Response: Thank you for this suggestion. We agree that behavioral preservation is the appropriate property to verify for the Neutralized and Randomized transformations. The original construction checks establish syntactic validity and dependency completeness, but do not independently establish behavioral equivalence. We have therefore added execution based validation of the transformed functions.

1. What the original checks guarantee.

A function is retained during dataset construction only if it parses, byte compiles, and resolves every free symbol in a pinned environment (Section 3.1.1). These checks ensure that the extracted snippets have the required dependencies, including library calls, helper definitions, and class definitions. Such dependencies are frequently missing from earlier code benchmarks (L54–56).

This issue is particularly relevant to R. A study of 9,078 R files from Harvard Dataverse found that 74% failed to complete without error during an initial execution [5].

BRACE is a retrieval benchmark, so no model generated code is executed. The relevance of each query code pair is instead validated through the expert hypothesis test (Table 2).

Related transformation based benchmarks also use different validation approaches. CLARC [1] filters for compilability, ReCode [2] assesses semantic preservation using human raters, with agreement exceeding 90% for perturbed prompts, and similarity scores, while CodeXGLUE AdvTest [4] normalizes names to func and arg_i. To our knowledge, these benchmarks do not report execution based verification of all transformed variants to establish unchanged behavior.

2. Additional execution based behavioral validation.

We now explicitly test whether each transformed function preserves the behavior of its original implementation.

Let ff denote the original function, TT the transformation, σ\sigma the identifier mapping, and π\pi the accessor mapping. For valid inputs xx, we test:

T(f)(σ(args),π(x))=π(f(x))T(f)(\sigma(args),\pi(x)) = \pi(f(x))

We compare the resulting values, attributes, classes, and raised conditions.

Inputs are traced from package examples, tests, and vignettes following the approach of genthat [3]. For functions without suitable traced inputs, we use documented fixtures, such as scuttle::mockSCE and DESeq2::makeExampleDESeqDataSet, and report these separately.

Table R2. Behavioral validation of transformed evaluation functions.

CheckResult
Executed on traced real inputs, including fixtures52% / 82%
All variants re pass parse, compile, and resolve checks100%
Equivalent, Neutralized, before / after fixes91% / 98%
Randomized, all 10 seedsSame as Neutralized
AST round trip, before / after printer fixes99.6% / 100%
Compiled Bytecode reproduces source outputs100%
Largest change in a Standard to anonymized gap, verified items2.0 NDCG points

The 52% figure represents functions executed using traced real inputs, while 82% includes the additional documented fixtures.


To be continued...

AuthorsOpenReview ↗

Response to Reviewer WU8b - Part 3

Continued after Part 2...


3. Failure analysis and corrections.

Most initial failures arose from schema positions that the renamer did not cover. Examples include a data.frame(padj = ...) tag or a string index in [ that remained unchanged while padj was renamed.

Other failures involved name reflective constructs, including:

  • deparse(substitute()), get, and exists.
  • do.call("name") and S3 dispatch by name.
  • Formulas and tidy evaluation masks.
  • $ partial matching.

We extended the mapping to cover these positions and removed the 20 functions, representing approximately 2% of executed items, that still failed behavioral validation.

The remaining 18% of functions could not be executed offline because they depend on external files, downloads, or graphics devices. These functions retain their construction level verification through parsing, compilation, and dependency resolution, and receive a static audit for the same name reflective constructs.

Among the verified items, the Standard to anonymized performance gaps in Table 4 change by at most 1.5 NDCG points. Therefore, the additional behavioral validation does not alter any of the our conclusions.

Manuscript revision. We will add the following description to Appendix, Verification of transformed code:

"We verify that the Neutralized, Randomized, AST, and Bytecode variants preserve the behavior of the original functions. For each function ff, with identifier mapping σ\sigma and accessor mapping π\pi, we execute T(f)(σ(args),π(x))T(f)(\sigma(args),\pi(x)) and compare its outputs with π(f(x))\pi(f(x)), using inputs traced from package examples, tests, and vignettes following genthat, or documented fixtures. We execute 82% of the evaluation functions, of which 98% are behaviorally equivalent after correcting the renaming procedure. Functions that still fail are removed. All retained variants pass the parse, compile, and dependency resolution checks."

The new validation establishes behavioral equivalence through execution for the functions that can be run offline, with a static audit for the remainder. The small change in retrieval performance gaps among verified functions further supports the validity of the reported anonymization results.

References

[1] Wang, Huang, Fang, and Wang. CLARC: C/C++ benchmark for robust code search. ICLR, 2026.

[2] Wang et al. ReCode: Robustness evaluation of code generation models. ACL, 2023.

[3] Krikava and Vitek. Tests from traces: automated unit test extraction for R. ISSTA, 2018.

[4] Lu et al. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. NeurIPS Datasets and Benchmarks, 2021.

[5] Trisovic, Lau, Pasquier, and Crosas. A large scale study on research code quality and execution. Scientific Data, 2022.


Response to Questions

Q3. Original BRACE versus anonymized Bytecode and AST on a common subset

Response: Thank you for this suggestion. The submitted tables already evaluate the same functions in the five representations: Standard in Table 3, Neutralized and Randomized in Table 4, and AST and Bytecode in Table 5. Both AST and Bytecode use the Neutralized identifier mapping (Section 3.1.3).

Consequently, comparing Neutralized with AST isolates the effect of changing the representation while holding the identifier mapping fixed. We bring these results together in Table R3 to make this comparison explicit.

Table R3. NDCG of the same functions across four representations (from Tables 3, 4, and 5). Group 3 refers to Group 3 Long, matching the configuration in Table 5.

GroupModelStandardNeutralizedASTBytecodeNeutralized to AST
1CodeT5+47.831.913.67.9−18.3
1Nomic85.982.647.829.4−34.8
1OASIS86.783.451.932.7−31.5
1OpenAI82.470.645.317.6−25.3
1Voyage87.684.161.538.9−22.6
2CodeT5+39.614.89.76.1−5.1
2Nomic90.867.938.926.2−29.0
2OASIS89.570.347.430.5−22.9
2OpenAI84.339.633.813.2−5.8
2Voyage92.176.856.237.1−20.6
3CodeT5+15.35.94.53.2−1.4
3Nomic66.835.224.113.6−11.1
3OASIS65.436.427.916.0−8.5
3OpenAI81.232.122.68.3−9.5
3Voyage87.060.742.721.4−18.0

To be continued...

Comments on the paper

7 comments
Area_Chair_rT9KOpenReview ↗

Discussion Period

Dear Reviewers,

The authors have posted their response. Please take a moment to review it and engage in discussion with both the authors and fellow reviewers, especially on any remaining concerns or points of disagreement. Brief updates to your assessment would be very helpful.

Thank you.

Dear Reviewers, Area Chairs, and People of the Internet,

I want to raise a concern about potential plagiarism. The current submission is very similar to our paper CLARC: C/C++ Benchmark for Robust Code Search, which was accepted by ICLR 2026 and posted on arXiv on March 4, 2026: https://arxiv.org/abs/2603.04484.

My small experiment identified substantial wording overlap:

Section in this submissionText covered by matches of at least 8 consecutive tokens
Related Works65.8% (506 / 769 tokens)
Experiment Setup43.0% (99 / 230 tokens)

These figures are based on normalized text that ignores capitalization, punctuation, and line breaks, with inline citations retained and overlapping matches counted only once. The longest contiguous match contains 117 normalized tokens.

Additionally, all six main-paper tables correspond to tables in CLARC in the same order. The authors also explicitly cite CLARC at line 131 of the current draft, demonstrating their awareness of the CLARC paper.

I respectfully ask the ACs to investigate this concern and, if necessary, escalate it to the Program Chairs. If misconduct is substantiated, I hope there would be appropriate sanctions beyond rejection. I also ask the chairs to disclose the authors’ identities even if this submission is withdrawn or desk-rejected, just like other rejected/withdrawn submissions at ICLR in previous years.

Sincerely,

Kaicheng

AuthorsOpenReview ↗

Response to the public comment on CLARC

We sincerely thank Kaicheng Wang for the careful comparison. Below we explain how the shared wording arose, what BRACE takes from CLARC, and what it contributes.


How the overlap arose

BRACE is designed as an R and biomedical follow-up to CLARC. It adopts CLARC's construction and evaluation protocol so that results can be compared across C/C++ and R. Because the two papers evaluate the same six models with the same metrics and discuss the same prior benchmarks, we drafted these background descriptions with CLARC as the reference paper, and some of its wording carried over. This applies to the descriptions of prior work, models, and metrics in Related Works, Experiment Setup, and the model appendix. It does not apply to the sections that report BRACE's design and results.

Using the same measurement as the comment (matches of at least 8 normalized tokens), the overlap with CLARC is 4.3% for Dataset (Section 3), 7.3% for Evaluation (Section 5), 3.2% for the Introduction, 1.7% for the Discussion and Conclusion, 0.0% for the Error Analysis appendix, and 0.0% for the new validation appendix.

The shared table order follows from the shared protocol. Every number in those tables is new R data.

We did not hide the relationship. Section 2 names CLARC as the most closely related work. During the rebuttal, before this comment, we told Reviewer SQQr that BRACE reuses the construction and validation protocol of CLARC.


What BRACE takes from CLARC

BRACE takes the following from CLARC, and we claim none of them as our contribution:

  • the pipeline stages: an executability filter, three dependency groups, LLM-written queries validated by the hypothesis test of Wang et al. (2023), and Neutralized and Randomized anonymization;
  • the model suite;
  • fine-tuning of CodeT5+ and OASIS with InfoNCE;
  • the order of sections and tables.

What BRACE contributes

ComponentCLARCBRACE
Language and sourceC/C++, GitHubR; 58 Bioconductor 3.22 packages and 86 biomedical GitHub repositories
ExecutabilityCompilationByte-compilation and resolution of every free symbol (codetools) in a pinned renv environment (R 4.5.1, Bioconductor 3.22)
Dependency groupsCustom types, helper functionsR class systems (S3, S4, R6, Bioconductor containers), helper functions
AnonymizationNeutralized, RandomizedNeutralized and Randomized, plus string-typed data accessors ($, [[, @, tidy evaluation) with an ablation
Lowered formsAssembly, WebAssemblyR bytecode (compiler::cmpfun), the form installed packages ship, and an S-expression AST setting
Error analysisNo domain-aligned error-analysis400 failures, 7 categories, Cohen's kappa 0.71
Behavioral validationCompilabilityExecution of original and transformed functions on traced inputs: 82% executed, 98% equivalent

Every query, snippet, annotation, and result in BRACE is new, and CLARC contains no R data from which they could be derived. The new components produce findings that the CLARC protocol alone could not:

  1. Accessor literals: intact string-typed accessors recover 41% to 59% (mean 52%) of the Group 2 neutralization NDCG loss. This channel exists because R addresses data fields by string.
  2. Conditional reliance: the code-specialized embedders lose 3.3 to 3.5 NDCG points on base-R functions under neutralization, and 15.3 to 22.9 points on container-class functions.
  3. Representation alone: the AST setting keeps the Neutralized identifier mapping, so it isolates representation. It costs 5.8 to 34.8 further NDCG points for the four models that support R.
  4. Language coverage: CodeT5+, whose pre-training excludes R, trails the R-capable code models by 38.1 to 39.8 NDCG points.
  5. Robustness gap is not data-limited: doubling the training data to 10,724 pairs changes the best-case robustness gap by at most 1.2 MRR points.

We apologize for the oversight in these background passages. The overlap is confined to descriptions of prior work, models, and metrics. The sections presenting our contributions share almost no wording with CLARC. BRACE is motivated by CLARC, which we explicitly identified as the closest related work. While its protocol is inherited, the benchmark, R specific settings, experiments, and findings are independently developed and constitute new work and evidence.

We disclosed the protocol reuse in our rebuttal before this concern was raised. We acted in good faith and acknowledge that this does not excuse the textual overlap, which we take responsibility for correcting. We respectfully ask the ACs and PCs to distinguish this writing oversight from an attempt to appropriate or misrepresent CLARC's scientific contributions.

We are rewriting Related Works, Experiment Setup, and the model appendix, and adding explicit attribution to CLARC's protocol in the abstract, contributions, and Section 3; and upload it soon.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.