- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether multiple accepted solutions and rejected submissions from the same programming problem can provide richer supervision for code retrieval than a standard single-positive contrastive learning baseline. It also compares adapted models against frozen pretrained encoders and examines whether the reported gains persist after removing documentation and comments. Overall, I find the problem setting reasonable and potentially interesting. However, in its current form, the paper is not yet sufficiently convincing in terms of novelty, empirical support, or presentation quality. The core idea appears closer to an application of existing multi-positive contrastive learning ideas to a specific code data structure than to a clearly new methodological contribution. In addition, the empirical gains are generally small and unstable, which makes the paper’s main claim difficult to evaluate with confidence. The presentation also remains a significant issue: after reading the abstract and introduction, I still found the paper’s research question, motivation, and intended contribution insufficiently clear.
Critical strengths
- The paper asks a reasonable research question. Using multiple accepted solutions and rejected submissions from the same problem as supervision signals is intuitively well motivated for code retrieval.
- The experimental evaluation is relatively careful. The paper does not only compare against a single-positive baseline, but also against frozen pretrained encoders, and it further checks whether the gains remain after removing documentation and comments. This is more convincing than a comparison limited to another finetuned baseline.
Critical weaknesses
- The writing is unclear, especially in the abstract and introduction. The paper does not clearly communicate its core research question, motivation, or why the proposed design choices are the right response to that motivation.
- The novelty is limited. The proposed approach mainly reads as an adaptation of standard multi-positive contrastive learning to the code retrieval setting, rather than a substantially new formulation or conceptual contribution.
- The empirical evidence is weak and not sufficiently robust. Many of the reported improvements are very small, and the paper does not provide enough ablation analysis to isolate which component actually drives the gains. As a result, the current results do not convincingly support a strong method claim.
Questions for authors
See above.