- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks whether program structure can identify internal states with corresponding roles across similar programs. PSCA extracts CFG/DFG relations, projects endpoints to token spans, and transfers complete hidden vectors through activation patching. Structure-aligned transfers are compared with same-offset, constrained-random, and clone-span transfers. The evaluation covers intact compatibility, post-disruption recovery, and restoration specificity across nine fine-tuned models from three model families.
Critical strengths
- The question is clear and productively separates output-level code similarity from internal computation.
- The empirical effort is substantial: CFG and DFG alignments, full layer sweeps, separate compatibility and recovery protocols, and within-pair controls. The authors also acknowledge that complete hidden states do not isolate role, lexical, or positional information.
Critical weaknesses
- The main effect is not isolated from surface confounds. Complete hidden vectors mix lexical, positional, and program-role information. Appendix F.3 gives an example in which a purely lexical coordinate can produce a positive compatibility effect. The paper includes clone-span and matched-generic controls, but these do not residualize lexical/positional subspaces or provide an exact nuisance-matched donor. The results support functional compatibility more strongly than correspondence of program-role representations.
- Training/evaluation independence is a central concern. Appendix A reports 3,030 training source functions and 3,030 selected greedy query records per model, 2,813 query families, and a library containing the originals of all 3,030 training functions. This strongly suggests substantial prompt/source reuse, and no disjoint held-out split is reported. Exact overlap statistics and a held-out rerun are needed before the effects can be interpreted as general correspondence rather than familiarity.
- Statistical dependence and the strongest baseline need more attention. The nine models share source prompts and control examples. Main effect intervals are pair-bootstrap, not query/source-family clustered; Appendix C's family resampling concerns the NLL/KL correlation rather than the primary intervals. C.8 evaluates a clean-state neural cosine selector on the same subsets as the same-offset/constrained-random comparisons (1,279 CFG and 3,621 DFG), not the full main-effect populations. Its NLL intervals are below zero in five of eighteen model-by-abstraction comparisons (three CFG and two DFG; three of nine configurations) and include zero in the other thirteen; KL is not similarly one-sided. The selector also reproduces PSCA's assignment in most cases, so it belongs in the main table with clustered uncertainty.
- Novelty needs sharper empirical positioning. The contribution is the integration of relation matching with activation patching. The closest code-specific work, Abdelsalam et al. (2026), already applies activation patching to programs, and alignment-learning methods such as DAS/HyperDAS already provide alternatives. The paper explains its distinction, but it does not empirically test what program-structure-derived alignment adds beyond these approaches.
Questions for authors
- What are the exact function- and prompt-level overlap counts between training and evaluation, and do the conclusions hold on a fully disjoint held-out set?
- Can the authors add lexical/positional residualization or a matched-donor control that changes structural correspondence while holding surface features fixed?
- Can the neural cosine comparator be included in the main comparison with query/source-family clustered confidence intervals?
- Can the authors describe the model in detail, the model size and series for easier reproduction. Qwen is not informative.
3 comments
Reply to Reviewer arPa (2/2)
W3/Q3 (B). Cosine baseline.
Answer: We added the Cosine baseline to the main comparison and report two sets of 95% confidence intervals, one clustered by target origin function and the other by retrieved source function. Given the same target positions, PSCA selects source positions from program structure alone (black-box selection) and Cosine uses activations from both programs (white-box selection). We compared the methods on 1,404 of 1,566 CFG pairs and 3,746 of 3,813 DFG pairs. During intact transfer, PSCA and Cosine selected exactly the same positions in 60.9–89.8% of DFG cases and 88.6–95.6% of CFG cases across the nine configurations. For each architecture, the table gives mean NLL and KL (nats/token) across the Original, Lexical and Structural configurations during intact DFG transfer.
| NLL ↓ / KL ↓ | PSCA | Cosine | Same-offset | Random |
|---|---|---|---|---|
| Gemma | 0.1802 / 0.1291 | 0.1708 / 0.1197 | 0.7522 / 0.7060 | 0.7296 / 0.6822 |
| Llama | 0.5467 / 0.0181 | 0.5459 / 0.0176 | 1.1201 / 0.6528 | 1.1244 / 0.6662 |
| Qwen | 0.5189 / 0.0450 | 0.5070 / 0.0321 | 1.1847 / 0.7788 | 1.1584 / 0.7579 |
Across all three architectures, Cosine gave slightly lower mean NLL and KL than PSCA. Lower NLL indicates better prediction of the target tokens, and lower KL indicates closer agreement with the clean target distribution. We also compared the methods within each configuration for CFG and DFG (18 comparisons per metric). With either grouping, most comparisons showed no clear difference between PSCA and Cosine during intact transfer. Where the difference was clear, NLL or KL was lower with Cosine. [§3.2, Table 1 and Appendices C.6–C.8]
Q4. Model series and sizes.
Answer: We use Gemma-2-9B-it, Llama-3.1-8B-Instruct and Qwen2.5-Coder-7B-Instruct. Base denotes the released instruction-tuned model. Original denotes the same model after we fine-tuned it on original code. Appendix A gives our fine-tuning settings.
W4. Contribution of program structure to state transfer.
Answer: Across all nine DFG configurations, NLL was 0.42–0.71 nats/token lower with PSCA than with same-offset or random selection. Abdelsalam et al. (2026) apply activation patching to program variants whose token positions are aligned by construction. PSCA identifies corresponding token positions by matching control-flow or data-flow relations between the target program and a retrieved source program. We kept the target positions the same across the three selection methods to measure what program structure contributes to source selection. [§3.2, Table 1]
On the structural binding benchmark, distributed alignment search (DAS) distinguished changes in binding more clearly at PSCA positions than at mismatch positions. PSCA selects the source positions, and DAS learns which subspace of the state to transfer. HyperDAS also learns which positions to use. We trained DAS separately at PSCA and mismatch positions. At both sets of positions, we measured the logit difference (LD) between scores for method names of the target and alternative types. The table shows how much more a transfer reduces LD when the source program changes the binding than when it preserves it. Each row averages the Base, Original, Lexical and Structural models of one architecture, using the same cases within each model for both columns (Table 35).
| Architecture | DAS at PSCA positions | DAS at mismatch positions |
|---|---|---|
| Gemma | 5.43 | 0.003 |
| Llama | 3.35 | −0.004 |
| Qwen | 5.91 | 0.000 |
At PSCA positions, changing the binding shifted the prediction further toward the alternative type in all three architectures. At mismatch positions, both transfers shifted the prediction, but the difference between the two source programs was close to zero. [§3.5 and Appendix G.4]
Dear Reviewer arPa,
Thank you again for your thoughtful review and constructive feedback on our submission.
As the reviewer–author discussion period approaches its conclusion, we would be grateful if you could revisit our responses. We hope our clarifications and additional experimental results have helped address your concerns, and we would be happy to provide further clarification or respond to any additional questions you may have.
Thank you very much for your time and consideration.
Best regards,
Author(s)
Reply to Reviewer arPa (1/2)
Answer: We added a matched-source control (your “matched donor”) to the structural binding benchmark. When both source programs use the same name at the final use, changing the binding shifts the target prediction further toward the alternative type than preserving it.
In the first source program, the renamed variable refers to the same assignment as in the target program. In the second source program, the same name refers to a different assignment. We transfer the state from each source program’s final variable use to the same position in the target program. We also keep the prediction position the same. The table shows how much more a binding change reduces the logit difference (LD) between the model’s scores for method names of the target and alternative types. Each row averages the Base, Original, Lexical and Structural models of one architecture (Table 29).
We also compared binding changes with name changes. The median LD reduction was 2.49 when we changed the binding, 0.05 when we renamed the queried variable, and below 0.01 when we changed only the names in the other two definitions. On the retrieved-program benchmark, we measure complete-state compatibility and restoration. [§3.5, Figure 5 and Appendices F.3, G.1]
Answer: We generated each target program in the main set from the task instruction of a function used in fine-tuning (the target’s origin function). Both the function and its instruction appeared in fine-tuning for all 819 CFG and 2,731 DFG query families (each origin counted once per configuration). The reformatted inference prompts had zero exact matches to the training instructions. To retrieve source programs, we built a database of 51,191 distinct Python functions from 120 GitHub repositories, including the originals of all 3,030 source functions used in fine-tuning. The retrieved source appeared in fine-tuning for 1,371 of 1,566 CFG pairs and 3,519 of 3,813 DFG pairs.
For the held-out set, we excluded fine-tuning functions and their close matches before selecting origin functions and retrieving source programs. The held-out target instructions had zero exact matches to the fine-tuning instructions. DFG compatibility and restoration specificity were positive for the Original and Base models of Gemma, Llama and Qwen. CFG restoration specificity was positive in three of these six models. In the main set, restoration specificity was positive in 9/9 DFG and 8/9 CFG configurations both for transfers from the target program’s origin function and for transfers from another function (non-origin). The table gives mean DFG advantages over mismatch transfer (NLL, nats/token). [§3.1, §3.4 and Appendices A.2–A.3, C.9–C.11]
Answer: Program pairs can share a target origin function or a retrieved source function. To account for this dependence, we estimated 95% confidence intervals by resampling groups of pairs. We grouped pairs by target origin in one analysis and by retrieved source in the other. With either grouping, DFG recovery and restoration specificity were significant in all nine configurations for both NLL and KL.
Counting each group once per configuration gives 819 query families and 973 source groups for CFG, and 2,731 query families and 2,870 source groups for DFG. We averaged cases within pairs and gave each pair equal weight. We tested each configuration separately and report the mean effects across all nine (NLL / KL, nats/token).
With either grouping, DFG compatibility was significant in eight of nine configurations for NLL and all nine for KL. For CFG recovery, the NLL effect was significant in three of nine configurations when grouped by target origin and four when grouped by source function. For KL, it was significant in seven of nine with either grouping. [§3.1, Appendix C, Table 7]