- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VANISH, a framework for cross-language code clone detection that combines features from the source code with features from an LLM-generated pseudo-assembly counterpart. With the complementary assembly view, VANISH aims to tackle the challenges caused by real-world code obfuscation. On XL-Asm, a new dataset crafted by the authors, VANISH shows higher detection accuracy and robustness compared with existing baseline models.
Critical strengths
- The paper targets a practical problem, cross-language code clone detection, which is important for code reuse and retrieval in software development.
- The authors find that the compiled representation of source code can serve as a complementary detection signal that improves performance under obfuscation, and propose a solution that combines source code and assembly code for detection.
- Experiments on a newly crafted multilingual dataset demonstrate the effectiveness of the proposed method compared with several existing baseline models.
Critical weaknesses
- It is unclear why the authors choose x86-64 pseudo-assembly as the only compiled representation in VANISH, rather than other representations such as LLVM IR, which existing studies [1-2] have already used for cross-language code matching. The authors should provide an ablation to justify this choice.
- The paper lacks justification for deliberately treating same-language clones as negatives during training.
- The experiments lack an LLM re-write attack. As more and more code is processed by LLM agents, the proposed method should be shown to be robust to LLM re-writes of the code in order to survive this real-world challenge.
- Potential comparison unfairness and selectively reported numbers. First, in Table 1 the baselines are used off the shelf while VANISH is fine-tuned on the proposed dataset, which makes the comparison unfair. Second, the AUC recovery reported in the abstract is measured on a subset rather than the full test set; the results on the full test set may be much lower.
- The writing and presentation of the paper could be improved. For example, Figure 1 should separate the dataset generation part and the VANISH training/deployment part into two figures for clarity.
[1] Gui, Yi, et al. "Cross-language binary-source code matching with intermediate representations." IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2022.
[2] TehraniJamsaz, Ali, Hanze Chen, and Ali Jannesari. "GraphBinMatch: Graph-based similarity learning for cross-language binary and source code matching." IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). 2024.
Questions for authors
- The full VANISH inference time is much longer than that of the source-code-only baselines. Did the authors try a smaller ViC model for faster assembly generation? Or is it possible to use existing compilation tools instead of the ViC model on the supported languages to reduce the inference cost?
7 comments
Gentle Follow-up
Dear Reviewer,
With the rebuttal period now more than halfway through, we wanted to check whether you have had an opportunity to review our responses and updated manuscript. The revised PDF includes our additional experiments and clarifications, with changes marked in blue.
We would welcome your feedback on whether these additions resolve the issues you raised, and we remain available to address any outstanding questions. If the new results and revisions have sufficiently addressed your concerns, we kindly ask you to consider increasing your score. We sincerely appreciate your time and careful consideration.
Thanks for the rebuttal from the authors. Here are some follow-up questions and suggestions:
W1: The truncation rates (89.8–98.8%) in the W1 experiments are quite high. Could the lack of fusion gains be due to information loss caused by truncation?
W3: Could the authors compare the proposed method and its variants with the other baselines reported in the paper on the same LLM-rewritten data? This would clarify whether VANISH still outperforms the other baselines despite its observed performance decline on rewritten data.
W4: Given the disclosed test-set exposure, the main results in Table 1 should be replaced with results obtained using models trained on test-disjoint data.
Response to Reviewers
Thank you for the review. All four experiments report null/negative findings. CIs: 95% paired task-cluster bootstrap (2,000 resamples), multiplicity-uncorrected.
A frozen source branch is fused with clang LLVM IR/native assembly or stored ViC. Auxiliary encoders share data/protocol; test: 254 C/C++ records/127 tasks. Source/auxiliary token limits: 256/512; truncation: 89.8–98.8%. Clean-dev weights favor source-only on ties. Scores: X-MRR pp.
Native assembly leads standalone: +8.32 [+2.78, +13.66] over ViC; +7.94 [+2.17, +13.22] over IR. IR−ViC: +0.37 [−4.47, +5.05] (ns). No view improves fusion on this narrow, truncation-limited cohort.
We claim toolchain-free availability, not superiority. Unlike XLIR/GraphBinMatch’s shared IR, virtual compilation needs no per-language compiler pipeline. Ours covers C/C++, not the other ten languages; audited clean parsing: 68–86%. This limits implementation, not compiler existence. Stored ViC coverage reflects construction, not 100% fresh-generation success.
Only the positive mask changed: cross-language vs. all same-task non-self positives. Training: 600 updates/arm, 2,098 groups; test: 15,711 records/3,728 tasks. X-MRR: 68.28/68.69, +0.41 pp [−0.17, +0.96]. Queries improved/harmed: 19.8%/17.4%.
Neither policy significantly leads. Cross-language positives are evaluation-aligned, not superior or universal in historical checkpoints. This single-seed, bounded-update test cannot establish converged equivalence.
One fixed-prompt Python rewrite/program; originals/rewrites tested on ≤20 official IO cases. Of 80 tasks, 60 originals passed, 50 pairs verified: 37 queries/179 candidates. ViC accepted: 31/37 queries, 149/179 candidates.
Clean→rewritten changes use 24 queries/149 candidates with common accepted views; fusion gains use 37 queries/179 candidates, full-gallery source fallback if either required auxiliary view is rejected. Clean-dev weights stay fixed.
Full 37-query source X-MRR: 53.58→53.92 (+0.34 pp; no paired CI). Common-cohort assembly: 25.00→9.40; R@1: 20.83→0.00. Rewriting exposes assembly sensitivity without measured source decline. Renaming robustness ≠ structural-rewriting invariance. One model/prompt/temperature-zero sample cannot characterize adaptive attackers; sampled IO agreement cannot prove semantic equivalence.
Adaptation: Matched budgets replace unqualified SOTA and +12.3%/+68.5% claims: same data, 1,200 updates, batch 16, seed 719, dev selection. Adapted CodeBERT/C4: 51.76/45.25 X-MRR vs. adapted CodeSage source’s 73.11. C4 adapts the objective, not the full system. Capacity: CodeSage ≈356M vs. CodeBERT-base ≈125M.
Scope: Subset AUC recovery is no general headline. Full clean test (15,711 records; 31,422 pairs): AUC 0.9900→0.9945; X-MRR +0.36 pp [+0.06, +0.65]. This is not obfuscated-subset recovery or proof of full-obfuscated-test robustness. Fresh clean record-level gains lose significance after program deduplication (H.3). Clean C/C++ parsing: 62/80 and 57/80 vs. perturbed 13/80 and 0/80. Older attacks are historical diagnostics, not verified semantics-preserving robustness evidence.
Provenance: Historical source training/checkpoint selection included 530/144 final-test groups, respectively; assembly selection used test X-MRR (H.2). This corrects assigning 530 groups solely to checkpoint selection. Affected results are qualified; dev-only selection cannot erase exposure. Fresh controls use audited, overlap-filtered training data and dev-selected checkpoints.
Figs. 1–2 separate construction from training/deployment; H reports cohort sizes/generation-success counts.
Smaller ViC is untested. Sequential uncached probe median latency: 27.81 s for 9 successes; 32.75 s for all 10 attempts vs. source-only retrieval’s 0.022 s. Median paired ratio: 1,473×. Clang assembly p50: 0.049/0.403 s for dev C/C++; 0.786 s for test C++. Lower cost/stronger standalone W1 results motivate compiler-first processing with virtual fallback where unsupported; this hybrid is untested. Neither real nor virtual views improve cohort fusion: cheaper generation ≠ retrieval benefit.
We added representation/positive-mask ablations and IO-tested rewriting, clarified scope/provenance/costs, and split construction/deployment diagrams (H.2–H.7; §§4.1–4.2, 4.5–4.6; Figs. 1–2).