acceptodds
Under review as a conference paper at ICLR 2027

Did the Model Read the Paper? Attributing Paper-to-Code Scores to Their Sources

Abstract

Paper-to-code benchmarks are meant to measure whether models can turn research papers into working code, but their scores may also reflect information already available in the provided repository. We study this source-attribution problem in ResearchCodeBench, where only the target code span is masked while the other annotated code in the same file remains visible. We introduce paired interventions that separately remove the paper and this neighboring code while keeping the task and tests fixed. Across three models, removing the paper reduces pass rate by 1.6–8.1 percentage points (nominally significant for one model), whereas masking the neighboring code reduces it by 16.8–25.4 points and eliminates roughly two thirds of successful cases; most of this effect remains when the masked neighbors are replaced by a list of the names they bind. The asymmetry is concentrated in short targets nested inside another annotated snippet. A pre-specified split of the mask shows that most of it comes from the rest of that enclosing snippet, which the benchmark leaves visible, while the method's other components matter no more than the paper. The asymmetry also shrinks under the benchmark's line-weighted metric. We further find that naively masking multiple implementations makes the original prompt ambiguous, inflating the measured effect of code context. We fix this evaluation artifact by explicitly naming the target to restore, which recovers 6.5–8.6 points, while the larger effect of neighboring code remains. Our results show that repository-assisted paper-to-code benchmarks should control alternative implementation sources, starting with the code that encloses the target, before interpreting test-passing code as evidence that a model implemented the paper.

Primary area:
datasets and benchmarks
Keywords:
paper-to-code generation, benchmark validity, code LLM evaluation, source attribution, context leakage, ablation, ResearchCodeBench

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wBwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether passes on ResearchCodeBench reflect use of the paper or of code already visible in the prompt. On 186 runnable tasks with fixed tests and grader, it uses paired prompt interventions: removing or replacing the paper, and masking the other annotated snippets in the target file, with length controls and a names-only condition. Removing the paper lowers pass rate by 1.6 to 8.1 points, while masking the neighbors lowers it by 16.8 to 25.4 points. Most of the latter comes from the enclosing snippet of nested targets. The paper also shows that naive masking makes the original instruction ambiguous, and fixes this by naming the target. Results are checked on six endpoints, with repository-level inference, and on a second benchmark.

Critical strengths
  1. The paired design is careful. Task, tests and grader stay fixed, and the length-matched wrong paper, names-only condition and length controls address the obvious confounds. This makes the attribution claims credible within RCB.
  2. The instruction artifact is a concrete and transferable finding. Any ablation that changes the number of visible placeholders can silently change the measurement, and the paper shows how to detect and fix it.
Critical weaknesses
  1. The main finding is narrow and partly expected. Most of the asymmetry comes from nested targets where the rest of the enclosing function stays visible, which is a construction choice of one benchmark, and prior work such as CrossCodeEval and SpecAgent already shows that surrounding code strongly helps completion. The SciReplicate-Bench check fails the authors' own pre-specified criterion, and the fully masked variant is too small to say whether the paper matters once this leakage is removed. The paper therefore reads more as a diagnostic note on RCB than as a general result on paper-to-code evaluation.
  2. The paper-side estimate, which is the core question in the title, is fragile. Its size is close to the measured grading noise, and its significance changes with the serving engine and the unit of inference. Under RCB's line-weighted metric, the neighbor effect on Gemma is no longer significant. Nesting and target length can not be seperated, the CONTROL condition is not reproducable, and one main model is non-deterministic and no longer served. The caveats are reported honestly, but together they leave the headline less firm than the abstract suggests.
Questions for authors
  • If targets are defined at the outermost annotated unit on a larger task pool, does the paper effect become measurable, and does the asymmetry with the neighbor effect remain?
  • Is there a design that separates nesting from target length, for example by matching nested and outermost targets on reference length?
5 comments
AuthorsOpenReview ↗

Response to Reviewer wBwy (1/5)

Response to Reviewer wBwy

We thank the reviewer for recognizing the careful paired design and the importance of the instruction artifact. We agree that the large neighbor effect observed on RCB, particularly for nested targets, should not be generalized to paper-to-code benchmarks without further evidence. In response, we have conducted additional experiments to examine the contribution of enclosing code while holding the target fixed, assessed the effects of target length and evaluation uncertainty, and clarified the scope of our findings. We also emphasize the broader methodological contribution: successful code restoration does not identify which information source supported it, and interventions designed to measure source dependence can themselves introduce evaluation artifacts.

W1 and Q1. What can we conclude on outermost targets and beyond RCB?

The main finding is narrow and partly expected. Most of the asymmetry comes from nested targets where the rest of the enclosing function stays visible, which is a construction choice of one benchmark, and prior work such as CrossCodeEval and SpecAgent already shows that surrounding code strongly helps completion. The SciReplicate-Bench check fails the authors' own pre-specified criterion, and the fully masked variant is too small to say whether the paper matters once this leakage is removed. The paper therefore reads more as a diagnostic note on RCB than as a general result on paper-to-code evaluation.

If targets are defined at the outermost annotated unit on a larger task pool, does the paper effect become measurable, and does the asymmetry with the neighbor effect remain?

We agree that surrounding code helping completion is already established, and we do not claim otherwise. Nor do we claim that neighboring implementations generally contribute more than papers. Our central question is different: a successful implementation does not identify which supplied information source enabled it, and interventions intended to measure this dependence can themselves introduce evaluation artifacts. We address this through controlled source comparisons, analysis of task construction, and checks of intervention validity.

First, the asymmetry is not established on RCB's outermost targets. We examined the 60 existing outermost targets, where no enclosing annotated implementation remains visible.

Table R11. Pass rate changes on RCB's outermost targets (pp). Brackets give joint task bootstrap 95% confidence intervals. A positive gap means masking neighbors causes a larger loss.

Endpoints averagedRemove paperMask neighborsAdditional masking loss
Three main models−6.1 [−12.8, 0.3]−11.2 [−20.6, −2.8]5.1 [−4.5, 15.1]
All six endpoints−5.8 [−10.0, −1.6]−12.8 [−20.6, −5.8]7.0 [−0.6, 15.1]

The six-endpoint average suggests a measurable paper contribution under task-level resampling, although these endpoints share the same tasks. More importantly, neither estimate establishes that neighbor masking costs more than paper removal on outermost targets. On Qwen, paper removal actually costs 13.3 pp, compared with 10.0 pp for neighbor masking, although their difference is not significant. We therefore restrict the large asymmetry to the RCB tasks where it is supported, particularly nested targets.

Redefining targets at the outermost annotated unit yields only 65 runnable parent tasks across 18 projects (one more project than our main analysis; Appendix D.2 explains why). Our fully masked variant cannot resolve a small paper effect at this scale. It also retains paper-derived specifications, so adding the paper measures its incremental contribution rather than its full contribution. We release this variant as a resource without drawing conclusions about paper dependence on a larger pool. On SciReplicate-Bench, the neighbor effect did not meet our prespecified replication criterion; the task construction differences that may explain the smaller effect are summarized in our response to Reviewer eugy (W4 and Q4).

AuthorsOpenReview ↗

Response to Reviewer wBwy (2/5)

Second, we examined whether the underlying source-attribution issue extends beyond RCB. Our structural audit of MMSciCode (ACL 2026), an independently constructed benchmark with 624 function-level tasks, found that other targets' reference implementations remain visible in 74.5% of its prompts under the released builder's default configuration (35.6% within the same file). Among the 259 Python tasks with another target in the same file, only one (0.4%) is nested. Thus, potential alternative implementation sources are not unique to RCB's nested annotations. However, their presence alone does not establish that models depend on them.

The same audit shows that removing the paper does not remove all information derived from the paper or the reference implementation. On these 259 tasks, NO-PAPER deletes the paper excerpts, the quoted passage, the governing formula and the symbol mapping, but keeps a method description written by the benchmark authors and a "Return contract" field that the builder extracts from the reference implementation. This field appears in 77.6% of tasks, and in 49.4% it contains more than a variable name or constant, for example return np.maximum(0, x). A paper-removal comparison on MMSciCode therefore measures the incremental contribution of the removed fields, not the full contribution of the paper. NO-PAPER prompts also still instruct the model to "honor the paper's formulas and the provided symbol mapping" after both have been removed. This is the same class of instruction mismatch as in our third finding below. Our own NO-PAPER condition on MMSciCode keeps this instruction, as pre-registered, so its paper-removal estimate should be read as an incremental contribution measured under this mismatch.

We therefore ran the same paired interventions on these 259 MMSciCode tasks, with Gemma-4-12B-it and Qwen2.5-Coder-32B (one greedy run each) and Claude-Haiku-4.5 (majority of three runs). Prompts, tests and per-sample environments are MMSciCode's own. Its Dockerfiles install from PyPI only and miss the CPU builds of PyTorch and some CUDA libraries, so we added these before grading any model, with the same environments for every condition. Because of this subset, these environments, the excluded tasks below, our single-prompt setup and our models, our pass rates are not comparable with the accuracy MMSciCode reports; only the paired differences between conditions are interpreted. Before generation, we fixed the analysis: the neighbor contrast (CONTROL → MASKED, length-matched) uses the tasks where masking removes a visible same-file implementation, and replication requires the predicted direction with p < 0.05 on at least two models. Following MMSciCode's own grading, a response without extractable code counts as a failure; in our RCB analyses such outputs are excluded, and the main RCB comparison is unchanged when they are scored as failures (Appendix A.5). Tasks whose reference solution fails in this setup (62 of 259, 57 of them in the neighbor subset) are dropped from every condition, leaving 132 tasks for the neighbor contrast and 197 for paper removal.

Masking same-file neighbors lowered pass rates by 9.1 pp on Gemma (95% CI [2.8, 11.7], McNemar p = 0.004), 2.3 pp on Qwen ([−4.0, 7.7], p = 0.61) and 10.6 pp on Haiku ([4.2, 13.3], p = 0.001). Two models meet the pre-specified criterion, so the neighbor effect replicates on MMSciCode; Qwen's interval includes zero. Because several tasks come from the same paper, we additionally resampled the 64 papers in this subset (10,000 paper-cluster bootstrap resamples, as for RCB's repository clusters); this check was not part of the pre-specified analysis. The losses on Gemma (9.1 pp [2.9, 16.4]) and Haiku (10.6 pp [4.7, 17.2]) remain above zero, and Qwen's interval still includes zero (2.3 pp [−3.1, 8.3]). We have added this check to Appendix F.1. Removing the paper changed pass rates by −1.5, −1.5 and −3.0 pp (all p ≥ 0.24). This is the incremental contribution described above, measured under the instruction mismatch and on a different task set, so we do not compare it with the neighbor effect.

AuthorsOpenReview ↗

Response to Reviewer wBwy (3/5)

Table R12. Task construction on MMSciCode. Same definitions as Table R10. The MMSciCode column is the subset where masking removes a visible same-file implementation, before excluding tasks whose reference solution fails. Table 18 in the revision reports this column next to all 259 same-file tasks (where masking removes a median of 0.7 times the target's length).

Prompt propertyRCB (186)SciReplicate, same-file (59)MMSciCode, masked subset (189)
Targets nested inside another annotated span67.7%0%0.5%
Other annotated targets in the same file, median1321
Reference solution length, median lines3.51416
Visible lines removed / target lines, median11.2×1.7×1.5×
Masking removes an identifier used by the target78.0%37.3%32.8%

MMSciCode targets are whole functions that are almost never nested, and in this subset masking removes a median of 1.5 times the target's length in visible code. Hiding these same-file implementations still lowered pass rates on two of three models. Together with RCB's outermost targets, where masking neighbors cost 11.2 pp (Table R11), this indicates that visible neighboring implementations can serve as an alternative source outside nested annotations. The point estimates are below RCB's overall neighbor effect, but the contrasts and task sets differ, so we read this only descriptively and do not rank benchmarks by effect size. All three models were evaluated on the same 132 tasks, so Qwen's smaller and non-significant effect cannot be explained by task construction alone. A secondary contrast that also masks cross-file implementations, for which we set no criterion, was in the same direction but not significant (p = 0.12, 0.61 and 0.064). These results also do not show that neighbors contribute more than the paper on MMSciCode, for the reasons given above.

Third, the contribution extends beyond the magnitude of the RCB neighbor effect. Our study provides three findings:

  1. Source attribution: Paired interventions compare the contributions of papers and visible implementations while keeping the target, tests, and grader fixed. This distinguishes successful task completion from evidence that a particular source was necessary for success.
  2. Construction diagnosis: We identify the enclosing implementation as the dominant source of the RCB neighbor effect, accounting for 76–84% of the separated masking losses on eligible nested targets. This shows how masking granularity changes the information available for restoration.
  3. Intervention validity: Masking multiple snippets makes the original single-target instruction ambiguous. Naming the target restores 6.5–8.6 pp on the submitted endpoints (6.5–9.7 pp including the replacement Mistral endpoint), demonstrating that an ablation can change the measurement procedure rather than only remove information.

The last finding is especially relevant beyond RCB: any evaluation that changes the number or identity of missing targets must ensure that the instruction and answer extraction still measure the intended task. The source-attribution comparisons and construction checks likewise provide a procedure that other benchmarks can adopt, without assuming that they will reproduce RCB's effect sizes.

Accordingly, we have revised the abstract, introduction, and conclusion to lead with source attribution, task construction, and intervention validity; the comparison of paper and code remains as one result, restricted to RCB and concentrated in nested targets, not as a general ranking of their importance. A new Section 4.5 (Table 5) brings the cross-benchmark construction check and the pre-specified neighbor contrasts on SciReplicate-Bench and MMSciCode into the main text, as a check suggested by three benchmarks rather than a validated predictor; Section 4.4 is condensed. Appendix F.1 now includes the cross-benchmark structural audit and practical checks for identifying alternative implementation sources. It now also reports the MMSciCode interventions, including the excluded tasks and the environment additions.

Reviewer_eugyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether successful code restoration in paper-to-code benchmarks stems from scientific papers or from pre-existing implementation context within the repository. Using ResearchCodeBench (RCB), the authors employ paired interventions that remove the paper or mask neighboring implementations while holding the target and evaluation procedure constant. Across three principal models, removing the paper reduces pass rates by 1.6–8.1 percentage points (pp), while masking neighboring implementations causes a much steeper 16.8–25.4 pp drop. The authors trace this asymmetry primarily to nested targets, where the enclosing implementation leaks substantial information. They additionally uncover a critical evaluation artifact: masking multiple code spans creates ambiguous placeholders, and explicitly naming the target restores 6.5–8.6 pp of the pass rate. The paper addresses a critical construct-validity question with rigorous controls, and my remaining feedback focuses on refining the boundaries of its claims regarding external validity and evaluation metrics.

Critical strengths
  • The paper draws a highly valuable distinction between a model successfully generating code and a model actually utilizing the scientific paper to do so. The use of paired interventions, precise length controls (e.g., matching character counts for paper deletion), and rigorous statistical sensitivity analyses greatly bolsters the empirical findings.

  • By identifying the enclosing implementation as the dominant driver of the neighbor effect, the authors demystify the underlying mechanism behind the performance drop.

  • The authors exhibit commendable academic honesty, appropriately contextualizing metric dependence, repository-level uncertainty, and the statistical limitations of their supplementary tests.

Critical weaknesses
  • Table 2 demonstrates that the neighbor-versus-paper difference is −21.6 pp for nested targets, but drops to an unestablished −5.1 pp for outermost targets. Furthermore, 76–84% of the neighbor effect on eligible nested targets originates from the enclosing implementation rather than from independent algorithmic components. While the authors appropriately acknowledge this, it structurally limits broader interpretations regarding a model's reliance on independent neighboring implementations. Additionally, nesting and target length are nearly collinear, complicating the separation of their effects. To better separate the variables of nesting, length, and context, consider a controlled experiment that varies the amount of visible enclosing code for targets of comparable lengths.

  • The headline results utilize unweighted task-level pass rates, whereas RCB's leaderboard employs a line-weighted scaled pass@1 metric. Under this weighting, the neighbor-masking effect shrinks considerably. For Gemma, the effect is no longer statistically significant, and for Qwen, the gap between paper removal and neighbor masking narrows to just 2.7 pp. While the unweighted pass rate effectively illustrates task-level dependence, the official metric is important when discussing the practical interpretation of benchmark scores. Report the direct paired paper-versus-neighbor comparisons under the official metric, complete with repository-cluster confidence intervals, alongside the primary unweighted results in the main text.

  • The authors transparently report that masking removes identifiers used by the target in 145 of 186 tasks, and that 107 tasks still lack these identifiers under the Names-Only condition. Consequently, the measured implementation contribution cannot be interpreted strictly as algorithmic dependence. Introduce an interface-preserving ablation that retains necessary signatures, dependencies, and structural information while exclusively stripping the algorithmic bodies. Even a manually validated sample would dramatically strengthen the source-attribution claim.

  • Based on Appendix F.1 (Table 13), elaborating on the differences between RCB's single-file span restorations and SciReplicate-Bench's function-level agentic reproduction would provide valuable guidance on when repository context is likely to leak vs. when it remains insulated.

Questions for authors
  • Under the benchmark's official line-weighted metric, is the paired difference between paper removal and neighbor masking statistically significant for each main model after applying repository-level clustering?

  • Since the removal of full implementations also removes structural cues, can you quantify how much of the implementation effect remains if function signatures, required imports, and dependency interfaces are preserved, while exclusively masking the algorithmic implementation bodies?

  • Why was CONTROL+NAMED not evaluated across all three principal models rather than relying on sensitivity analyses to bridge the instruction gap?

  • While the lack of statistical significance on SciReplicate-Bench is mathematically limited by the small number of discordant pairs, the absolute magnitude of the neighbor effect (5.4–6.9 pp) remains substantially smaller than in RCB. What specific task construction differences between the datasets (e.g., annotation nesting frequency, target length, repository structure) do you believe explain this reduced magnitude?

7 comments
AuthorsOpenReview ↗

Response to Reviewer eugy (1/5)

Response to Reviewer eugy

We thank the reviewer for the detailed assessment and constructive suggestions. We have conducted additional experiments to isolate the contribution of enclosing code, evaluated the direct comparison under RCB's official metric, and examined instruction effects and differences in benchmark construction. We also clarify the limitations of our current interface analysis.

W1. Distinguishing enclosing context from target length

To better separate the variables of nesting, length, and context, consider a controlled experiment that varies the amount of visible enclosing code for targets of comparable lengths.

We agree that nesting and target length are strongly associated in RCB. The original nested versus outermost comparison cannot establish their independent effects. Matching targets by reference length reduces the difference in neighbor-masking losses to 1.6–4.0 pp across the three main models, with all repository bootstrap confidence intervals including zero.

We therefore conducted an additional experiment that varies the visible enclosing code while keeping the target fixed. For each nested target, we retained all, the first half, the second half, or none of its enclosing implementation. The target, its length, other neighbors, instruction, and tests remained unchanged. Across the masking conditions, the paper also remained fixed. Each masking condition had a separate length control that retained the enclosing code while removing the corresponding number of characters from the paper.

Table R6. Pass rate changes from masking enclosing code (pp). Results are relative to each condition's length control. Brackets give 95% repository bootstrap confidence intervals. The experiment covers 125 Gemma and 126 Qwen nested targets across 14 repositories.

Visible enclosing codeGemmaQwenAverage [95% CI]
None−19.2−21.4−20.3 [−31.6, −10.8]
First half−10.4−15.9−13.1 [−23.2, −6.0]
Second half−12.0−18.3−15.1 [−24.1, −8.2]

Removing the entire enclosing implementation reduced pass rates on both models (Holm-adjusted p = 0.007 each). Retaining half recovered an average of 6.2 pp [1.8, 10.0] relative to complete masking. These results show that visible enclosing code contributes to successful restoration even when the target and its length remain fixed. While the independent effect of nesting across different targets remains unresolved, the distinction between algorithmic information and interface cues is addressed separately in our response to W3/Q2.

Accordingly, we have added this controlled experiment to Table 2 and Appendix C.4 and revised our discussion to distinguish the contribution of visible enclosing code from the relationship between nesting and target length.

AuthorsOpenReview ↗

Response to Reviewer eugy (2/5)

W2 and Q1. Direct comparison under RCB's official metric

The headline results utilize unweighted task-level pass rates, whereas RCB's leaderboard employs a line-weighted scaled pass@1 metric. Report the direct paired paper-versus-neighbor comparisons under the official metric, complete with repository-cluster confidence intervals, alongside the primary unweighted results.

Under the benchmark's official line-weighted metric, is the paired difference between paper removal and neighbor masking statistically significant for each main model after applying repository-level clustering?

We agree that the magnitude and statistical support depend on how tasks are weighted. Equal task weighting measures the average change in success across implementation tasks, matching our question of how removing each information source affects task completion. This also follows the Pass@1 definition in the NeurIPS 2025 (Datasets and Benchmarks) version of the RCB paper (Sec. 2.2), which reports line weighting as an additional metric in Appendix I. The RCB website leaderboard, however, uses line weighting, giving longer reference implementations greater influence. We therefore report both metrics to distinguish their implications.

We computed the direct paired comparison between NO-PAPER and MASKED under the same NAMED instruction and task set, rather than comparing the significance of the two interventions separately. Under the line-weighted metric, only Mistral-Medium-3.5 shows a statistically significant difference after repository-level clustering.

Table R7. Additional loss from neighbor masking compared with paper removal (pp). Positive values indicate a larger loss from masking neighbors. Brackets give 95% repository bootstrap confidence intervals. Holm-adjusted p values are from cluster-robust t tests across the three models within each metric.

ModelEqual task weightingHolm pOfficial line weightingHolm p
Gemma-4-12B-it15.3 [4.9, 25.1]0.0163.9 [−5.7, 13.7]0.48
Mistral-Medium-3.5 (original endpoint)22.2 [11.9, 30.9]<0.0018.6 [3.8, 12.9]0.003
Qwen2.5-Coder-32B11.3 [3.1, 19.4]0.0162.7 [−2.7, 6.7]0.48

Under equal task weighting, the differences are positive on all three models, although the more conservative exact repository sign-flip test gives Holm-adjusted p values of 0.059, 0.021, and 0.059, respectively. We selected this family of three model comparisons after seeing the data, and the Table 4 caption now says so. In the pre-specified 39-test family (Table 16), which corrects across all of our contrasts, the equal-weight gap survives with task-level p values on Gemma and Mistral, and with repository-level p values on Mistral alone (cluster-robust t test) or on none (sign flip). We therefore treat the size of this gap as a task-level description on RCB rather than a general ranking of the two sources, and Section 4.1 states both families. Under the official metric, only Mistral's interval excludes zero. The other two intervals do not establish the difference, but also do not establish its absence.

The reduction under line weighting reflects the task structure: neighbor masking has its largest effects on short targets, whereas the longest third of targets accounts for 76% of the official metric's weight. Thus, the two metrics answer different questions about task-level dependence and leaderboard score changes.

Accordingly, we have added the direct paired comparisons under both metrics to Table 4 in the main text and revised our discussion to distinguish task-level source dependence from effects on the official leaderboard score.

AuthorsOpenReview ↗

Response to Reviewer eugy (3/5)

W3 and Q2. Separating implementation content from dependency interfaces

The measured implementation contribution cannot be interpreted strictly as algorithmic dependence. Introduce an interface-preserving ablation that retains necessary signatures, dependencies, and structural information while exclusively stripping the algorithmic bodies.

Can you quantify how much of the implementation effect remains if function signatures, required imports, and dependency interfaces are preserved?

We agree that NAMES-ONLY does not preserve complete dependency interfaces. Among 186 runnable tasks, masking removes an identifier used by the target in 145 tasks, and 107 still lack such an identifier under NAMES-ONLY. Therefore, our original decomposition establishes that full implementations provide more information than bound names alone, but does not isolate algorithmic content.

We designed an additional interface-preserving ablation that retains imports, function and class signatures, required bindings, and dependency interfaces while masking implementation bodies. Under MASKED, each annotated snippet disappears entirely, including any function or class headers inside it. The interface-preserving condition instead keeps, within each masked snippet, import lines, decorators and def/class headers verbatim, the left-hand side of each assignment (x = ...), return ..., and comment lines listing the names the snippet binds, uses, and reads from outside; it removes right-hand sides and control flow. These lists are computed from the masked neighbor's own lines, never from the target. The one exception to removing control flow is the block enclosing the target, whose headers (for example if/else) stay so that the target keeps its position, so this condition does not remove the enclosing-snippet channel as completely as MASKED does. The masking rules use only information available in the original visible code, without consulting the target's reference solution. The target, paper, NAMED instruction, and grading tests remain fixed. Before generation, two annotators checked a sample of 48 tasks for whether the retained text leaks the algorithm and whether it suffices as an interface; no item failed (95% upper bound 6.05% on the per-task leak rate), after two items were adjudicated as code the benchmark itself already shows. The retained name lists can mention operations a neighbor applies (for example torch.tanh) without showing how they combine; any leak the audit missed would shrink the interface-preserving loss and bias the share toward the interface reading, which matters most for Qwen's low share.

Table R8. Pass rate losses after preserving dependency interfaces (pp). Losses are computed against length-matched controls on the same tasks. Brackets denote 95% repository bootstrap confidence intervals. The last row is the share of the full-masking loss that remains when interfaces are preserved. Gemma uses the 180 tasks graded under all four conditions, so its full-masking loss differs slightly from the 14.4 pp reported in our response to Q3.

InterventionGemmaQwenMistral (new endpoint)
Interface-preserving masking−11.7 [−19.7, −3.8]−3.8 [−9.4, 1.9]−14.0 [−22.7, −5.7]
Full neighbor masking−15.0 [−24.6, −6.2]−19.9 [−28.5, −11.0]−24.7 [−34.6, −13.6]
Share of the loss retained0.78 [0.43, 1.10]0.19 [−0.14, 0.40]0.57 [0.29, 0.96]

How much of the loss remains with interfaces preserved depends on the model. On Gemma, the point estimate indicates that most of the loss remains, pointing to implementation bodies, although its interval extends down to 0.43. On Qwen, preserving interfaces recovers most of the loss, and the remaining 3.8 pp is not distinguishable from zero. Mistral falls in between. Before generation, we fixed how this share would be read: 0.75 or more indicates mainly implementation content, 0.25 or less indicates mainly interface information, and the claim would be revised if two of the three models fell at or below 0.25. Only Qwen did. With 17 repositories, the intervals are wide, so none of these shares is precise. The visible reference code therefore supplies both interface information and implementation content, in proportions that vary across models, and we do not attribute the neighbor effect to algorithmic content alone.

Accordingly, we have added the interface-preserving analysis to Section 4.3 and Appendix C.3 (Table 9, with a prompt example and the audit), revised the abstract and Section 4.3 to state that visible reference code supplies both interface information and implementation content in model-dependent proportions, and revised our discussion to distinguish the contribution of implementation bodies from the interface information removed by the original masking procedure.

Reviewer_2NWROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies source attribution in ResearchCodeBench (RCB), a paper-to-code benchmark in which a model receives a paper plus a source file with one annotated span masked. The key observation is that RCB masks only the target span, leaving reference implementations of other annotated spans ("neighbors") in the same file visible. Through paired prompt interventions (remove the paper; mask the neighbors) with fixed tests and grader, the authors show across three models that removing the paper costs 1.6–8.1 pp of pass rate while masking neighbors costs 16.8–25.4 pp. A pre-specified decomposition attributes 76–84% of the neighbor effect on nested targets to the rest of the enclosing snippet that RCB's one-span-at-a-time masking leaves visible. The authors also identify a subtle evaluation artifact: naive neighbor masking creates multiple TODO placeholders under an instruction written for one, inflating the measured effect by 6.5–8.6 pp; naming the target corrects this. A fully masked variant of RCB is released with documented limitations.

Critical strengths
  1. Well-motivated construct validity question.

  2. Careful experimental design.

  3. The instruction artifact is a genuinely novel finding.

Critical weaknesses
  1. The one nominally significant paper effect (Qwen2.5-Coder-32B) flips significance depending on the serving engine (FP8: −5.4 pp, p=0.121; BF16: −8.1 pp, p=0.0135).

  2. Metric dependence undercuts the primary contrast. Under RCB's own line-weighted leaderboard metric, the neighbor effect shrinks to less than half and becomes non-significant on one model.

  3. One benchmark family (17 repositories), mostly mid-tier endpoints, and the pre-registered replication on SciReplicate-Bench fails its own criterion.

Questions for authors
  1. Can you regenerate CONTROL with a fixed seed across all endpoints and report whether Figure 4 and Table 3 (lower block) change, particularly for Mistral-Medium-3.5?

  2. Why does CONTROL exceed OFFICIAL by 4.3 pp on Mistral-Medium-3.5? Does cutting arbitrary paper text help, or is this noise? This bears on whether your length controls are actually neutral.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2NWR (1/5)

Response to Reviewer 2NWR

We thank the reviewer for recognizing the motivation behind our study of paper use in ResearchCodeBench (RCB) and the care we took in designing the paired experiments. We also appreciate the reviewer’s recognition of a novel finding: leaving the target unspecified among several TODO blocks inflates the apparent cost of masking neighboring code.

W1. Sensitivity of the paper effect to the serving configuration

The one nominally significant paper effect (Qwen2.5-Coder-32B) flips significance depending on the serving engine (FP8: −5.4 pp, p=0.121; BF16: −8.1 pp, p=0.0135).

We agree that the nominal significance of Qwen2.5-Coder-32B’s paper effect depends on the KV cache configuration. Removing the paper lowered its pass rate by 5.4 percentage points under FP8 and 8.1 points under BF16, but only the latter reached nominal significance. We retain both estimates and do not interpret the non-significant FP8 result as evidence that the paper provides no benefit.

We directly compared the two interventions under both Qwen configurations using the existing runs. The submitted paper already evaluated OFFICIAL, NO-PAPER and MASKED under both configurations with the corrected masker and NAMED instruction (Appendix F.3). For the additional analysis below, we restricted the comparison to the 186 tasks with verdicts for all three conditions under both configurations. The gap is NO-PAPER minus MASKED: a positive value means that masking neighbors causes a larger loss than removing the paper.

Table R1. Qwen performance gaps across KV cache configurations (pp). Brackets indicate 95% repository bootstrap CIs.

Effect on Qwen (percentage points)BF16 KV cacheFP8 KV cacheFP8 minus BF16
Additional loss from neighbor masking11.3 [3.1, 19.4]13.4 [4.4, 22.9]2.2 [−0.9, 5.5]

All three estimates use the same task subset. Confidence intervals are obtained by resampling repositories, using the same resamples for both configurations and their paired difference. Under both configurations, masking neighbors caused a larger loss than removing the paper, and both intervals exclude zero. The direct comparison is therefore supported under each configuration by these intervals, even though the significance of the paper effect itself changes between them; the more conservative repository sign-flip test is reported in our response to W2. The gap was 2.2 pp larger under FP8, but this interval includes zero. We therefore do not identify a clear change in the gap between configurations, which is not the same as showing that the configuration has no effect. Per-task verdicts agreed between the two configurations on 95.7–97.3% of tasks in each condition.

Additional results on Gemma-4-12B-it under both KV cache settings. We additionally evaluated Gemma-4-12B-it under BF16 and FP8 KV caches for OFFICIAL, NO-PAPER and MASKED. Both configurations used the same server and container image, with all serving arguments held fixed except for the KV cache precision.

Table R2. Gemma performance gaps across KV cache configurations (pp). Brackets indicate 95% repository bootstrap CIs, without adjustment for multiple comparisons.

Effect on Gemma (percentage points)BF16 KV cacheFP8 KV cache
Paper removal (NO-PAPER minus OFFICIAL)−1.6 [−7.2, 4.1]+0.6 [−5.7, 6.3]
Additional loss from neighbor masking (NO-PAPER minus MASKED)15.3 [4.9, 25.1]16.6 [6.3, 27.7]

Gemma’s estimated paper effect changed sign, with both intervals including zero, whereas the direct gap between the interventions was positive under both configurations, with both intervals excluding zero. On the 179 tasks graded under both configurations, the paired change in the paper effect was +2.2 percentage points [−4.4, 7.4]. This interval cannot exclude a shift as small as Qwen’s 2.7 points. Thus, this Gemma experiment supports the direct source comparison under both tested settings.

Accordingly, we have added the Gemma comparison to Appendix F.3 and revised our interpretation to emphasize the magnitude and uncertainty of the paper effect rather than whether it reaches statistical significance. We have also added Table R1 to Appendix F.3. We also clarify that conclusions about the relative effects of the two interventions rest on their direct comparison within each configuration, not on the absence of a significant paper effect.

AuthorsOpenReview ↗

Response to Reviewer 2NWR (2/5)

W2. What changes under the leaderboard metric?

Metric dependence undercuts the primary contrast. Under RCB's own line-weighted leaderboard metric, the neighbor effect shrinks to less than half and becomes non-significant on one model.

We agree that the magnitude and statistical support depend on how tasks are weighted. Equal task weighting measures the average change in success across implementation tasks, matching our question of how removing each information source affects task completion. This also follows the Pass@1 definition in the NeurIPS 2025 (Datasets and Benchmarks) version of the RCB paper (Sec. 2.2), which reports line weighting as an additional metric in Appendix I. The RCB website uses line weighting, giving longer reference implementations more influence. In the revision, we report both to distinguish their implications.

On Gemma, masking neighbors lowered pass rate relative to OFFICIAL by 16.8 pp with equal task weights. The 15.3 pp result in Table R3 and the 14.4 pp result in Table 3 instead use NO-PAPER and length-matched CONTROL as baselines, respectively. Under line weighting, the change relative to OFFICIAL was −3.7 pp (95% task bootstrap CI [−10.0, +3.4]). The largest masking losses occurred on short targets, whereas the longest third of targets received 76% of the total line weight. Line weighting therefore reduces the contribution of the short targets most affected by masking, helping explain the smaller aggregate masking effect.

Table R3. Additional loss from neighbor masking under two evaluation metrics (pp). Unlike the changes above, which are relative to OFFICIAL, this table compares MASKED with NO-PAPER. Brackets indicate 95% repository bootstrap CIs.

ModelEqual weight per taskLine weightingHolm p, line weighting
Gemma-4-12B-it15.3 [4.9, 25.1]3.9 [−5.7, 13.7]0.48
Mistral-Medium-3.5 (original endpoint)22.2 [11.9, 30.9]8.6 [3.8, 12.9]0.003
Qwen2.5-Coder-32B11.3 [3.1, 19.4]2.7 [−2.7, 6.7]0.48

With equal task weights, the repository cluster t test supported a larger loss from neighbor masking on all three models. The exact repository sign flip test gave Holm p = 0.059 for Gemma, 0.021 for Mistral, and 0.059 for Qwen, so statistical support also depends on the test. We selected this family of three model comparisons after seeing the data. In the pre-specified 39-test family (Table 16), which corrects across all of our contrasts, the gap survives with task-level p values on Gemma and Mistral, and with repository-level p values on Mistral alone (cluster-robust t test) or on none (sign flip); Section 4.1 now states this. This is why the revision no longer presents the ranking of paper and neighbors as its main result: the conclusions we rely on are that a pass cannot be attributed to the paper without controlling for visible implementations, the same-target enclosing-code experiment (Holm-adjusted p = 0.007 on each model; response to Reviewer eugy, W1), and the instruction artifact. The larger gaps under equal weighting therefore describe differences in average task performance; they do not establish a significant gap across all three models under the leaderboard weighting.

AuthorsOpenReview ↗

Response to Reviewer 2NWR (3/5)

W3. Model coverage and replication on SciReplicate-Bench

One benchmark family (17 repositories), mostly mid-tier endpoints, and the pre-registered replication on SciReplicate-Bench fails its own criterion.

The submission already evaluated three additional endpoints on the same 186 RCB tasks (Appendix F.5), including two with baseline pass rates above 70%.

Table R4. Baseline pass rates and intervention losses on additional RCB endpoints. Losses are measured in pp.

ModelBaseline pass rateLoss from paper removal (pp)Loss from neighbor masking (pp)
Claude-Haiku-4.550.0%11.332.3
Claude-Sonnet-570.4%4.343.5
GPT-6-Astra80.1%1.632.8

Masking neighbors caused a larger loss on all three endpoints, while the paper effect varied substantially across them. These results broaden model coverage, but all use the same 17 repositories and therefore do not establish generalization across benchmarks.

A post hoc direct comparison under the official line weighting also found larger losses from masking: 9.3 pp on Haiku, 32.4 pp on Sonnet, and 23.8 pp on GPT-6-Astra (repository cluster-robust t tests with Holm correction across these three endpoints: p = 0.011, 0.009 and 0.011). The new experiments with regenerated CONTROL on these endpoints are reported in our response to Q1.

The SciReplicate-Bench evaluation did not meet our prespecified replication criterion. We required the neighbor effect to have the predicted direction and reach p < 0.05 on at least two models. On the pre-specified subset with another annotated task in the same file, masking neighbors cost 5.4 pp on Gemma, 5.4 pp on Qwen, and 6.9 pp on Haiku. None reached significance, and we do not count the consistent direction alone as successful replication.

Our structural audit found that 126 of 186 RCB targets are nested within another annotated unit, leaving the rest of the enclosing implementation visible. None of the 82 SciReplicate tasks examined had this structure. This difference is consistent with the smaller neighbor effects, but does not establish their cause. We also clarify that our SciReplicate evaluation used a single prompt rather than its original agentic pipeline.

A third benchmark. To check whether RCB's construction is unusual, we audited the prompts of MMSciCode (ACL 2026; 624 function-level tasks from 285 papers), released after RCB by a different group. Its released prompt builder masks one target function at a time and keeps the other functions of the file at their reference bodies. Under its default configuration, another target's reference body is visible in 74.5% of tasks (35.6% counting only the same file). Its targets are whole functions with a median of one other target per file, which is closer to SciReplicate than to RCB. We then ran the paired interventions on its 259 Python tasks with another target in the same file. The neighbor effect met the same pre-specified criterion that SciReplicate-Bench missed, with point estimates below RCB's on a different contrast and task set (details in our response to Reviewer wBwy, W1).

Accordingly, the revision includes the additional endpoint results in Figure 2(a), documents the structural comparison (now including MMSciCode) and SciReplicate evaluation protocol in Appendix F.1. We restrict the relative ranking of paper and neighboring code contributions to RCB, while separately reporting the neighbor masking effect observed on MMSciCode. The paired interventions and instruction checks can be applied to other benchmarks, but the relative contributions of paper and code must be established separately in each setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.