- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies functional memorization in code language models, where models reproduce the behavior of training functions despite low textual overlap. It proposes an agent-based execution framework and evaluates the phenomenon across four open-source models and five programming languages, together with proxy detection methods, duplication analyses, and post-training mitigation methods.
Critical strengths
- The research topic is interesting and valuable.
- The paper presents a clear direction for auditing code-model memorization beyond textual overlap and provides evidence across multiple models and programming languages.
Critical weaknesses
-
The current experiments are closer to an audit of known training samples, representing a somewhat limited range of real-world leakage scenarios. The experiments provide the model with the exact function signature and up to 250 tokens of preceding code, meaning that the auditor already possesses substantial context about the target code.
-
Differences between the reference model and the target model cannot be attributed solely to the target function’s exposure during training. The paper constructs approximate counterfactual controls using earlier and later training checkpoints. The two models differ by an entire training phase rather than only by the specific function being audited. The target model might generate a functionally similar completion more easily because of an overall improvement in coding capability.
Questions for authors
-
With a target of 10 agent-generated inputs per function, achieving 80% line coverage does not guarantee that critical branches and edge cases are tested. A small number of hard-coded rules might pass these inputs, while complex functions could be misclassified because of uncovered paths. How do the authors address the risk of misclassification caused by untested branches and edge cases?
-
Approximately 10% of functions fail to receive a conclusive score, and this figure rises to about 20% for Rust. Functions with complex dependencies, extensive state, or structures resembling real-world projects may be more likely to be excluded. The memorization risk associated with these functions may differ from that of the retained samples. Could the authors compare the length, number of dependencies, and code complexity of conclusive and inconclusive samples, and report the range of results obtained when treating inconclusive samples as either positive or negative cases?
-
The proxy detection experiments evaluate functional similarity rather than the full definition of counterfactual functional memorization. These experiments use a binary label based solely on whether the execution similarity between the target-model generation and the original function reaches 0.8. They do not incorporate the reference-model comparison or the low-textual-similarity condition into this label. How should these proxy results be interpreted in relation to the paper’s definition of counterfactual functional memorization?
-
The cross-agent consistency of the execution-based evaluations is insufficient to demonstrate the stability of the final memorization rate. Although the correlations between the continuous scores produced by different agents range from approximately 0.74 to 0.82 in Figure 8, the paper’s final conclusions rely on binary labels derived from a threshold of 0.8. High correlation can still mask substantial reclassification of samples near the threshold. Could the authors report label agreement across agents, variations in the final memorization rates, and fluctuations observed when the same samples are evaluated repeatedly?
-
The experiments exploring the relationship between semantic duplication and functional similarity with low textual overlap support only a correlational relationship. The number of semantic near-duplicates may covary with factors such as function popularity, code length, project scale, degree of templating, and frequency of occurrence during training. The existing logistic regression analyses do not adequately control for these factors. The paper proposes semantic deduplication as a mitigation strategy, but the supporting evidence remains incomplete. Could the authors conduct matching analyses or control for additional function attributes? A more direct validation would involve retraining small-scale models on corpora with and without semantic deduplication and comparing their functional memorization rates.
-
The reported memorization rates lack sufficient uncertainty estimates. Functional memorization rates range from only 0.3% to 3.4%, and the number of positive instances is small in some settings, yet Figure 2 does not provide confidence intervals. Generation uses a single greedy completion, while execution testing involves agent-based randomness. Consequently, observed differences across models, languages, and model scales may be affected by measurement variability. How do the authors assess the uncertainty in the reported memorization rates and the differences across models, languages, and model scales?
-
The post-training experiments demonstrate output suppression on known samples, which cannot be directly equated with forgetting the underlying logic. NPO and DPO are trained on the 275 identified functions, and the reduction in memorization is subsequently measured on those same functions. The model might simply stop generating the original implementation from the same prefixes while retaining the associated logic. What evidence distinguishes output suppression on these known samples from forgetting the underlying logic?
-
The description of the DPO-rewrite method is inconsistent. The main text states that the method uses “functionally equivalent rewrites” as preferred completions, whereas the prompt in the appendix requires code with different functionality and the filtering criteria require an execution similarity score of no more than 0.2. These settings represent fundamentally different training objectives. Could the authors reconcile the main text, prompt, and experimental description, and clarify which construction was used to produce Figures 13 and 14?
3 comments
Response to review (2/3)
report the range of results obtained when treating inconclusive samples as either positive or negative cases?
Between 6.7% and 10.3% of functions are inconclusive (and 20.1% for Rust). We deliberately leave them out of the memorization rates, as we prefer to report only functions for which the execution test gives reliable evidence. Counting all inconclusive functions as not memorized would give 0.9–3.1% instead of the reported 1.0–3.4%. Counting all of them as memorized would give 9.6–21.8%, yet this is unlikely to be a fair approximation.
For a more principled estimate, we label the inconclusive functions with our LLM judge (LLaMA-3.3-70B, "Ours"). We set the judge's threshold on the conclusive functions so that its FPR against the execution labels is at most 5%. An inconclusive function then counts as memorized if the LLM judge marks the target as equivalent, not the reference, and BLEU < 0.75. This gives rates close to those we report:
| Rate (%) | Python (CC) | Python (SE) | C | Java | Rust | Go |
|---|---|---|---|---|---|---|
| Reported (conclusive only) | 1.0 | 3.3 | 3.4 | 1.7 | 3.4 | 1.7 |
| All inconclusive = not memorized | 0.9 | 3.1 | 3.0 | 1.5 | 2.7 | 1.6 |
| All inconclusive = memorized | 10.5 | 9.6 | 12.3 | 11.7 | 21.8 | 11.1 |
| Inconclusive labelled by LLM judge | 1.1 | 3.2 | 3.2 | 1.7 | 3.0 | 1.7 |
How should these proxy results be interpreted in relation to the paper’s definition of counterfactual functional memorization?
Measuring SIM_func through execution based testing is the only expensive part of computing functional memorization, and we do it for both the target and reference model. Section 5.2 therefore evaluates how well cheaper metrics approximate this step, while condition (2) in Section 3 is a cheap textual check.
To test the proxies end-to-end, we also applied our LLM judge (LLaMA-3.3-70B, "Ours") to the reference model's generations and computed a judge-only counterfactual memorization rate: the judge marks the target as equivalent, not the reference, and BLEU < 0.75. The judge's threshold is calibrated on held-out functions so that it flags as many generations as the execution test does.
| Python (CC) | Python (SE) | C | Java | Rust | Go | |
|---|---|---|---|---|---|---|
| Execution-based rate (%) | 1.0 | 3.3 | 3.4 | 1.7 | 3.4 | 1.7 |
| LLM-judge-only rate (%) | 0.7 | 2.7 | 3.3 | 1.4 | 3.3 | 2.1 |
| Cohen's κ, judge vs. execution label per function | 0.44 | 0.32 | 0.33 | 0.38 | 0.19 | 0.31 |
As expected with the threshold calibration, the judge reproduces the aggregate rates well. Per function, however, its agreement with execution is limited (κ = 0.19–0.44), partly because its errors on the target and reference generations compound. We note that this is a first, untuned instantiation and expect a reliable end-to-end judge would require careful iteration on the model, prompt and thresholds, which we see as a promising direction for future work.
report label agreement across agents, variations in the final memorization rates, and fluctuations observed when the same samples are evaluated repeatedly?
As mentioned in Appendix D, we also ran the execution-based test with four different models and agentic harnesses: Claude Code with Opus-4.7 (main), Opus-4.6 and Sonnet-4.6, and Codex with GPT-5.5. For every pair we now compute Cohen's κ between their labels on the functions that both found conclusive, for the target label (S_target ≥ 0.8) and for the counterfactual memorization label.
| Agents | Functions | κ, target label | κ, counterfactual memorization label |
|---|---|---|---|
| Opus-4.7 vs. Opus-4.6 | 4,369 | 0.76 | 0.77 |
| Opus-4.7 vs. Sonnet-4.6 | 4,183 | 0.77 | 0.73 |
| Opus-4.7 vs. Codex GPT-5.5 | 4,272 | 0.69 | 0.63 |
| Opus-4.6 vs. Sonnet-4.6 | 4,378 | 0.73 | 0.73 |
| Opus-4.6 vs. Codex GPT-5.5 | 4,480 | 0.76 | 0.64 |
| Sonnet-4.6 vs. Codex GPT-5.5 | 4,273 | 0.74 | 0.65 |
The agreement is substantial (κ = 0.63–0.77) and the highest among the Claude Code agents. It increases with stricter thresholds: for the target label, pairwise κ rises from 0.57–0.72 at τ = 0.5 to 0.77–0.83 at τ = 1.0.
The resulting counterfactual functional memorization rates are (with 95% Wilson confidence intervals).
| Agent | Conclusive functions | Memorization rate % [95% CI] |
|---|---|---|
| Claude Code, Opus-4.7 (main) | 90.2% | 0.98 [0.73, 1.31] |
| Claude Code, Opus-4.6 | 94.6% | 0.93 [0.69, 1.25] |
| Claude Code, Sonnet-4.6 | 90.1% | 0.82 [0.60, 1.13] |
| Codex, GPT-5.5 | 93.3% | 0.54 [0.36, 0.79] |
To measure run-to-run variation of a single agent, we also re-ran the main agent (Claude Code, Opus-4.7) on 500 Python (CraneCode) functions for OLMo-3-32B: all 44 memorized ones and 456 random others. The two runs agree closely (κ = 0.83 for the memorization label), and the disagreements stem mostly from borderline scores.
Overall, while the absolute rate varies somewhat with the agentic harness and the run, we find functional memorization to be detected consistently.
Response to review (3/3)
Could the authors conduct matching analyses or control for additional function attributes?
We agree that our logistic regression may omit relevant covariates, so it supports only an association and not a causal link. Adding covariates that we can easily compute (such as function length) could reduce confounding but not really eliminate it. As any causal claim would ultimately require controlled experiments that retrain models on corpora with and without semantic deduplication, as the reviewer suggests, we leave this to future work.
We will rephrase Section 5.3 to describe the observed association without implying causality and to present semantic deduplication as a hypothesis to test rather than a recommended mitigation.
The reported memorization rates lack sufficient uncertainty estimates.
We already provide ablations for the randomness from the agentic execution, including different similarity thresholds (Figure 7), models and harnesses (Figure 8).
Yet, we now also report uncertainty for all rates. We split the execution-tested functions of each setup into 5 disjoint random subsets and report the mean ± standard deviation across splits, averaged over 100 random partitions. We also report 95% Wilson confidence intervals on the full-sample rate.
| Subset | OLMo-3-32B | OLMo-3-7B | StarCoder-15B | CrystalCoder-7B |
|---|---|---|---|---|
| Python (CC) | 1.0 ± 0.3 [0.7, 1.3] | 0.7 ± 0.3 [0.5, 1.0] | ||
| Python (SE) | 3.3 ± 0.5 [2.8, 3.8] | 2.2 ± 0.4 [1.8, 2.7] | ||
| C | 3.4 ± 0.6 [2.9, 3.9] | 2.8 ± 0.5 [2.3, 3.3] | ||
| Java | 1.7 ± 0.4 [1.4, 2.1] | 1.5 ± 0.4 [1.2, 1.9] | ||
| Rust | 3.4 ± 0.8 [2.7, 4.2] | 2.3 ± 0.7 [1.7, 3.0] | ||
| Go | 1.7 ± 0.5 [1.3, 2.2] | 1.0 ± 0.4 [0.7, 1.4] | ||
| Python | 0.3 ± 0.2 [0.2, 0.5] | 0.5 ± 0.2 [0.3, 0.8] |
Our main findings hold under this uncertainty. All intervals exclude zero, so functional memorization occurs in every setup. Stack-Edu rates exceed CraneCode rates for both OLMo models, with non-overlapping intervals. OLMo-3-32B also has a higher point estimate than OLMo-3-7B on all six subsets, although the intervals overlap everywhere except Python (Stack-Edu). We will add these intervals to Fig. 2.
What evidence distinguishes output suppression on these known samples from forgetting the underlying logic?
We explicitly refrained from using the term “unlearning/forgetting” when applying post-training methods as mitigation techniques. We believe our method does not adequately remove knowledge about training data functions from the model, but rather teaches it to suppress output. We will change the corresponding framing.
Could the authors reconcile the main text, prompt, and experimental description, and clarify which construction was used to produce Figures 13 and 14?
Sec. 5.3 indeed does not use the correct wording, and Appendix I is correct: the preferred completions should be functionally valid but functionally different rewrites, filtered to BLEU ≤ 0.3 and execution similarity ≤ 0.2. Figures 13 and 14 use this construction. A functionally equivalent preferred completion would reinforce exactly the logic we aim to suppress. We will fix Sec. 5.3 accordingly.
Response to review (1/3)
We thank the reviewer for their feedback and time spent on the paper. We now answer the questions and weaknesses raised and aim to update the PDF by the end of the discussion phase.
We agree that our setup assumes a "strong" auditor with access to the exact context preceding the function. This is a deliberate design choice: a model developer or data owner checking whether functional logic leaks should estimate the worst case, and this is the standard "discoverable" memorization setup in the extraction literature (Carlini et al., 2023; Ippolito et al., 2023; Nasr et al., 2025). This does not imply that the same rates would hold in real-world leakage scenarios, where users or adversaries typically have less context. We will further clarify this in Section 6.
Thank you for raising this important point. It is something we acknowledge in the limitations (Section 6, "our counterfactual is not a perfect leave-one-out version of the target…"), but are happy to discuss further.
In short:
For more details on each of these points, we refer to our rebuttal to reviewer nPAM.
We agree that measuring functional equivalence through execution comes with its limitations. It depends on the diversity and coverage of the inputs the agent generates, and a generation that reproduces only some execution paths may still leak meaningful logic. This can lead to false positives or false negatives, while the true label is also nuanced and likely context dependent. We acknowledge this in Section 6, noting that there is likely no single correct threshold, and we report ablations across thresholds (Fig. 7) and across models and agentic harnesses (Fig. 8).
We have now also added two direct checks:
We compared conclusive and inconclusive functions (OLMo-3-32B) on length (lines of code), code complexity (cyclomatic complexity) and number of dependencies (distinct functions called):
Inconclusive functions are only marginally longer and are not more complex, but they do call somewhat more external functions. This suggests that a function ends up inconclusive because its dependencies are harder to control.