acceptodds
Under review as a conference paper at ICLR 2027

Beyond Single Positives: An Empirical Audit of Solution-Set Adaptation for Code Retrieval

Abstract

Competitive-programming problems provide multiple accepted solutions and rejected submissions for the same task. We audit whether adapting code encoders with this solution-set supervision improves retrieval beyond a single-positive baseline and the original pretrained encoder without adaptation (the frozen encoder). We evaluate MPR-Hybrid (Multi-Positive Retrieval Hybrid), a training system that trains language-specific LoRA adapters with a Multi-Positive Pooled Contrastive (MPPC) loss, linking each problem query to multiple accepted solutions and sampled rejected submissions selected from candidate pools using submission verdicts and embedding similarity. We compare it with a single-positive in-batch contrastive-learning system trained with the InfoNCE loss, using the other examples in each batch as negatives on six encoder backbones, trained with CodeContests supervision and evaluated on its separate test set, with zero-shot CodeSearchNet transfer, a fixed pool of code candidates selected by keyword matching (BM25), including lexically similar incorrect distractors, to test retrieval when keyword overlap is misleading, and CodeXGLUE AdvTest. We also evaluate the original pretrained encoders without fine-tuning as frozen references on the available CodeContests and CodeSearchNet evaluations, to assess whether adaptation improves retrieval beyond the pretrained representation. The observed benefits vary across encoder backbones and evaluation protocols. On the fixed CodeContests official-test protocol for unseen problems, MPR-Hybrid has higher average scores for some backbones, but no result meets our two-metric confirmation rule for a MPR-Hybrid win. In zero-shot CodeSearchNet Java transfer, where candidate code almost never contains the query text, MPR-Hybrid beats the single-positive baseline on three of six backbones, but none improves on its frozen encoder under the same two-metric rule. On Python, every protocol with a MPR-Hybrid win also contains the query text verbatim in candidate documentation. Removing documentation and comments leaves two confirmed MPR-Hybrid gains over the single-positive baseline: i) one other confirmed win becomes a loss, one formerly inconclusive comparison becomes a loss, and ii) two confirmed wins become inconclusive. The audit shows how several apparent gains change interpretation under frozen-encoder comparisons and documentation removal, while identifying settings where gains over the single-positive baseline remain. The comparison evaluates complete adaptation systems and therefore does not independently isolate the effects of the loss function, candidate sampling, or which encoder layers receive the LoRA adapters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Code retrieval, contrastive learning, multi-positive learning, parameter-efficient fine-tuning, empirical evaluation, evaluation bias, zero-shot transfer

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kM1DOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether multiple accepted solutions and rejected submissions from the same programming problem can provide richer supervision for code retrieval than a standard single-positive contrastive learning baseline. It also compares adapted models against frozen pretrained encoders and examines whether the reported gains persist after removing documentation and comments. Overall, I find the problem setting reasonable and potentially interesting. However, in its current form, the paper is not yet sufficiently convincing in terms of novelty, empirical support, or presentation quality. The core idea appears closer to an application of existing multi-positive contrastive learning ideas to a specific code data structure than to a clearly new methodological contribution. In addition, the empirical gains are generally small and unstable, which makes the paper’s main claim difficult to evaluate with confidence. The presentation also remains a significant issue: after reading the abstract and introduction, I still found the paper’s research question, motivation, and intended contribution insufficiently clear.

Critical strengths
  • The paper asks a reasonable research question. Using multiple accepted solutions and rejected submissions from the same problem as supervision signals is intuitively well motivated for code retrieval.
  • The experimental evaluation is relatively careful. The paper does not only compare against a single-positive baseline, but also against frozen pretrained encoders, and it further checks whether the gains remain after removing documentation and comments. This is more convincing than a comparison limited to another finetuned baseline.
Critical weaknesses
  • The writing is unclear, especially in the abstract and introduction. The paper does not clearly communicate its core research question, motivation, or why the proposed design choices are the right response to that motivation.
  • The novelty is limited. The proposed approach mainly reads as an adaptation of standard multi-positive contrastive learning to the code retrieval setting, rather than a substantially new formulation or conceptual contribution.
  • The empirical evidence is weak and not sufficiently robust. Many of the reported improvements are very small, and the paper does not provide enough ablation analysis to isolate which component actually drives the gains. As a result, the current results do not convincingly support a strong method claim.
Questions for authors

See above.

Reviewer_ysxsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether leveraging multiple accepted solutions improves code retrieval compared to conventional single-positive contrastive learning. The authors introduce MPR-Hybrid, a LoRA-based adaptation system that combines multi-positive contrastive learning with tiered hard-negative sampling and a memory bank. Experiments across six code embedding models and multiple benchmarks evaluate its effectiveness, generalization, and robustness.

Critical strengths

None

Critical weaknesses

First, the loss formulation in MPR-Hybrid is not the optimal one for multi-positive contrastive learning. The authors should check [1] for the optimal loss design. By switching to the optimal loss design, the conclusions in this paper may be changed. Second, some of the findings discussed in this paper are already well-known in the code retrieval area. For example, removing code documentation eliminates or reverses several observed improvements.

[1] Khosla, Prannay, et al. "Supervised contrastive learning." Advances in neural information processing systems 33 (2020): 18661-18673.

Questions for authors

I am confused about experimental settings. It seems that the authors not only apply multiple positive examples in the proposed method but also apply a hard negative mining algorithm. So do the experimental results faithfully reflect the impact of contrastive learning with multiple positive examples? The author may need to clarify this point.

Reviewer_cNViOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether exploiting the solution-set structure of competitive-programming data improves text-to-code retrieval. The proposed MPR-Hybrid system combines multiple accepted solutions, verdict-based rejected submissions, tiered negative sampling, a memory bank, MPPC contrastive learning, and language-specific LoRA adaptation. It is compared against a single-positive InfoNCE LoRA baseline across six backbones. A central empirical finding is that many gains over the single-positive adapted baseline do not translate into gains over the frozen encoder, and several Python gains disappear or reverse when documentation/comments are removed.

Critical strengths
  1. The evaluation covers multiple backbones and several substantially different retrieval conditions. The study uses six encoders and evaluates unseen CodeContests problems, zero-shot CodeSearchNet transfer, BM25-selected hard candidates, identifier obfuscation, and documentation removal, rather than relying on a single benchmark.
  2. The authors find that positive Python candidates frequently contain the query verbatim in documentation, and removing documentation changes four of six CodeSearchNet Python comparisons, including converting some gains into losses or inconclusive results. This materially changes the interpretation of the headline transfer results.
  3. The paper is appropriately cautious about its own limitations. It explicitly acknowledges that its experiments compare complete systems, that only one training run is used, that Tier 3 is never actually sampled, and that some selected checkpoints occur during warmup.
Critical weaknesses
  1. The main comparison does not cleanly isolate the effect of solution-set adaptation. MPR-Hybrid and the single-positive baseline differ in multiple aspects, including positives, negative sampling, loss, temperature, batch construction, and LoRA targets, making causal attribution difficult.
  2. The CodeContests evaluation is not well aligned with the proposed supervision. It ranks many accepted solutions against only ten cross-problem negatives and does not directly test accepted versus rejected submissions for the same unseen problem.
  3. The documentation control is incomplete across robustness tests. BM25 and AdvTest still retain comments and docstrings, so their gains may also partially rely on natural-language overlap.
  4. The full three-tier negative-sampling design is not actually evaluated. Tier 3 is never selected during training, so its contribution cannot be assessed from the reported experiments.
  5. The statistical analysis does not capture training variance. Each adapter is trained only once, while the reported confidence intervals are obtained by bootstrapping evaluation queries, which may be insufficient for several very small performance differences.
Questions for authors

refer to the Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.