- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether passes on ResearchCodeBench reflect use of the paper or of code already visible in the prompt. On 186 runnable tasks with fixed tests and grader, it uses paired prompt interventions: removing or replacing the paper, and masking the other annotated snippets in the target file, with length controls and a names-only condition. Removing the paper lowers pass rate by 1.6 to 8.1 points, while masking the neighbors lowers it by 16.8 to 25.4 points. Most of the latter comes from the enclosing snippet of nested targets. The paper also shows that naive masking makes the original instruction ambiguous, and fixes this by naming the target. Results are checked on six endpoints, with repository-level inference, and on a second benchmark.
Critical strengths
- The paired design is careful. Task, tests and grader stay fixed, and the length-matched wrong paper, names-only condition and length controls address the obvious confounds. This makes the attribution claims credible within RCB.
- The instruction artifact is a concrete and transferable finding. Any ablation that changes the number of visible placeholders can silently change the measurement, and the paper shows how to detect and fix it.
Critical weaknesses
- The main finding is narrow and partly expected. Most of the asymmetry comes from nested targets where the rest of the enclosing function stays visible, which is a construction choice of one benchmark, and prior work such as CrossCodeEval and SpecAgent already shows that surrounding code strongly helps completion. The SciReplicate-Bench check fails the authors' own pre-specified criterion, and the fully masked variant is too small to say whether the paper matters once this leakage is removed. The paper therefore reads more as a diagnostic note on RCB than as a general result on paper-to-code evaluation.
- The paper-side estimate, which is the core question in the title, is fragile. Its size is close to the measured grading noise, and its significance changes with the serving engine and the unit of inference. Under RCB's line-weighted metric, the neighbor effect on Gemma is no longer significant. Nesting and target length can not be seperated, the CONTROL condition is not reproducable, and one main model is non-deterministic and no longer served. The caveats are reported honestly, but together they leave the headline less firm than the abstract suggests.
Questions for authors
- If targets are defined at the outermost annotated unit on a larger task pool, does the paper effect become measurable, and does the asymmetry with the neighbor effect remain?
- Is there a design that separates nesting from target length, for example by matching nested and outermost targets on reference length?
5 comments
Response to Reviewer wBwy (2/5)
Second, we examined whether the underlying source-attribution issue extends beyond RCB. Our structural audit of MMSciCode (ACL 2026), an independently constructed benchmark with 624 function-level tasks, found that other targets' reference implementations remain visible in 74.5% of its prompts under the released builder's default configuration (35.6% within the same file). Among the 259 Python tasks with another target in the same file, only one (0.4%) is nested. Thus, potential alternative implementation sources are not unique to RCB's nested annotations. However, their presence alone does not establish that models depend on them.
The same audit shows that removing the paper does not remove all information derived from the paper or the reference implementation. On these 259 tasks, NO-PAPER deletes the paper excerpts, the quoted passage, the governing formula and the symbol mapping, but keeps a method description written by the benchmark authors and a "Return contract" field that the builder extracts from the reference implementation. This field appears in 77.6% of tasks, and in 49.4% it contains more than a variable name or constant, for example return np.maximum(0, x). A paper-removal comparison on MMSciCode therefore measures the incremental contribution of the removed fields, not the full contribution of the paper. NO-PAPER prompts also still instruct the model to "honor the paper's formulas and the provided symbol mapping" after both have been removed. This is the same class of instruction mismatch as in our third finding below. Our own NO-PAPER condition on MMSciCode keeps this instruction, as pre-registered, so its paper-removal estimate should be read as an incremental contribution measured under this mismatch.
We therefore ran the same paired interventions on these 259 MMSciCode tasks, with Gemma-4-12B-it and Qwen2.5-Coder-32B (one greedy run each) and Claude-Haiku-4.5 (majority of three runs). Prompts, tests and per-sample environments are MMSciCode's own. Its Dockerfiles install from PyPI only and miss the CPU builds of PyTorch and some CUDA libraries, so we added these before grading any model, with the same environments for every condition. Because of this subset, these environments, the excluded tasks below, our single-prompt setup and our models, our pass rates are not comparable with the accuracy MMSciCode reports; only the paired differences between conditions are interpreted. Before generation, we fixed the analysis: the neighbor contrast (CONTROL → MASKED, length-matched) uses the tasks where masking removes a visible same-file implementation, and replication requires the predicted direction with p < 0.05 on at least two models. Following MMSciCode's own grading, a response without extractable code counts as a failure; in our RCB analyses such outputs are excluded, and the main RCB comparison is unchanged when they are scored as failures (Appendix A.5). Tasks whose reference solution fails in this setup (62 of 259, 57 of them in the neighbor subset) are dropped from every condition, leaving 132 tasks for the neighbor contrast and 197 for paper removal.
Masking same-file neighbors lowered pass rates by 9.1 pp on Gemma (95% CI [2.8, 11.7], McNemar p = 0.004), 2.3 pp on Qwen ([−4.0, 7.7], p = 0.61) and 10.6 pp on Haiku ([4.2, 13.3], p = 0.001). Two models meet the pre-specified criterion, so the neighbor effect replicates on MMSciCode; Qwen's interval includes zero. Because several tasks come from the same paper, we additionally resampled the 64 papers in this subset (10,000 paper-cluster bootstrap resamples, as for RCB's repository clusters); this check was not part of the pre-specified analysis. The losses on Gemma (9.1 pp [2.9, 16.4]) and Haiku (10.6 pp [4.7, 17.2]) remain above zero, and Qwen's interval still includes zero (2.3 pp [−3.1, 8.3]). We have added this check to Appendix F.1. Removing the paper changed pass rates by −1.5, −1.5 and −3.0 pp (all p ≥ 0.24). This is the incremental contribution described above, measured under the instruction mismatch and on a different task set, so we do not compare it with the neighbor effect.
Response to Reviewer wBwy (3/5)
Table R12. Task construction on MMSciCode. Same definitions as Table R10. The MMSciCode column is the subset where masking removes a visible same-file implementation, before excluding tasks whose reference solution fails. Table 18 in the revision reports this column next to all 259 same-file tasks (where masking removes a median of 0.7 times the target's length).
| Prompt property | RCB (186) | SciReplicate, same-file (59) | MMSciCode, masked subset (189) |
|---|---|---|---|
| Targets nested inside another annotated span | 67.7% | 0% | 0.5% |
| Other annotated targets in the same file, median | 13 | 2 | 1 |
| Reference solution length, median lines | 3.5 | 14 | 16 |
| Visible lines removed / target lines, median | 11.2× | 1.7× | 1.5× |
| Masking removes an identifier used by the target | 78.0% | 37.3% | 32.8% |
MMSciCode targets are whole functions that are almost never nested, and in this subset masking removes a median of 1.5 times the target's length in visible code. Hiding these same-file implementations still lowered pass rates on two of three models. Together with RCB's outermost targets, where masking neighbors cost 11.2 pp (Table R11), this indicates that visible neighboring implementations can serve as an alternative source outside nested annotations. The point estimates are below RCB's overall neighbor effect, but the contrasts and task sets differ, so we read this only descriptively and do not rank benchmarks by effect size. All three models were evaluated on the same 132 tasks, so Qwen's smaller and non-significant effect cannot be explained by task construction alone. A secondary contrast that also masks cross-file implementations, for which we set no criterion, was in the same direction but not significant (p = 0.12, 0.61 and 0.064). These results also do not show that neighbors contribute more than the paper on MMSciCode, for the reasons given above.
Third, the contribution extends beyond the magnitude of the RCB neighbor effect. Our study provides three findings:
- Source attribution: Paired interventions compare the contributions of papers and visible implementations while keeping the target, tests, and grader fixed. This distinguishes successful task completion from evidence that a particular source was necessary for success.
- Construction diagnosis: We identify the enclosing implementation as the dominant source of the RCB neighbor effect, accounting for 76–84% of the separated masking losses on eligible nested targets. This shows how masking granularity changes the information available for restoration.
- Intervention validity: Masking multiple snippets makes the original single-target instruction ambiguous. Naming the target restores 6.5–8.6 pp on the submitted endpoints (6.5–9.7 pp including the replacement Mistral endpoint), demonstrating that an ablation can change the measurement procedure rather than only remove information.
The last finding is especially relevant beyond RCB: any evaluation that changes the number or identity of missing targets must ensure that the instruction and answer extraction still measure the intended task. The source-attribution comparisons and construction checks likewise provide a procedure that other benchmarks can adopt, without assuming that they will reproduce RCB's effect sizes.
Accordingly, we have revised the abstract, introduction, and conclusion to lead with source attribution, task construction, and intervention validity; the comparison of paper and code remains as one result, restricted to RCB and concentrated in nested targets, not as a general ranking of their importance. A new Section 4.5 (Table 5) brings the cross-benchmark construction check and the pre-specified neighbor contrasts on SciReplicate-Bench and MMSciCode into the main text, as a check suggested by three benchmarks rather than a validated predictor; Section 4.4 is condensed. Appendix F.1 now includes the cross-benchmark structural audit and practical checks for identifying alternative implementation sources. It now also reports the MMSciCode interventions, including the excluded tasks and the environment additions.
Response to Reviewer wBwy (1/5)
Response to Reviewer wBwy
We thank the reviewer for recognizing the careful paired design and the importance of the instruction artifact. We agree that the large neighbor effect observed on RCB, particularly for nested targets, should not be generalized to paper-to-code benchmarks without further evidence. In response, we have conducted additional experiments to examine the contribution of enclosing code while holding the target fixed, assessed the effects of target length and evaluation uncertainty, and clarified the scope of our findings. We also emphasize the broader methodological contribution: successful code restoration does not identify which information source supported it, and interventions designed to measure source dependence can themselves introduce evaluation artifacts.
W1 and Q1. What can we conclude on outermost targets and beyond RCB?
We agree that surrounding code helping completion is already established, and we do not claim otherwise. Nor do we claim that neighboring implementations generally contribute more than papers. Our central question is different: a successful implementation does not identify which supplied information source enabled it, and interventions intended to measure this dependence can themselves introduce evaluation artifacts. We address this through controlled source comparisons, analysis of task construction, and checks of intervention validity.
First, the asymmetry is not established on RCB's outermost targets. We examined the 60 existing outermost targets, where no enclosing annotated implementation remains visible.
Table R11. Pass rate changes on RCB's outermost targets (pp). Brackets give joint task bootstrap 95% confidence intervals. A positive gap means masking neighbors causes a larger loss.
The six-endpoint average suggests a measurable paper contribution under task-level resampling, although these endpoints share the same tasks. More importantly, neither estimate establishes that neighbor masking costs more than paper removal on outermost targets. On Qwen, paper removal actually costs 13.3 pp, compared with 10.0 pp for neighbor masking, although their difference is not significant. We therefore restrict the large asymmetry to the RCB tasks where it is supported, particularly nested targets.
Redefining targets at the outermost annotated unit yields only 65 runnable parent tasks across 18 projects (one more project than our main analysis; Appendix D.2 explains why). Our fully masked variant cannot resolve a small paper effect at this scale. It also retains paper-derived specifications, so adding the paper measures its incremental contribution rather than its full contribution. We release this variant as a resource without drawing conclusions about paper dependence on a larger pool. On SciReplicate-Bench, the neighbor effect did not meet our prespecified replication criterion; the task construction differences that may explain the smaller effect are summarized in our response to Reviewer eugy (W4 and Q4).