- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigated whether LLM can be finetuned on decompiled code. The authors first prepared a dataset, DECOMPBENCH, consisting 660k source/decompiled code from Gentoo package tree (~2k projects). Then, fine-tune the LLMs LLM with GRPO using a decompilation-quality reward. The results show that
- Cross checkpoint gives better reconstruction, but not all samples are gaining improvements
- Better individual reconstructions do not necessarily yield better test scaling, and highlight the importance of evaluating candidate generation and selection separately.
Critical strengths
The authors introduced a well-documented dataset, and in this work authors group also by package version to mitigate data leakage between training and evaluation splits. This makes it a valuable data resource for the decompilation.
GED serves as an effective optimization metric for training LLMs on decompilation tasks, minimizing GED indicates generated code better aligned with original source.
Critical weaknesses
Whether LLMs can reconstruct source code from compiled binaries is already recognized feasible and is an active research area. Finetuning LLMs for decompilation has been studied extensively since LLM4Decompile, and other works such as Idioms and SK2Decompile have also explored this direction.
The dataset contribution is also limited. Several datasets and benchmarks already exist for this task, including DecompileBench, Decompile-Bench, and the proposed dataset does not appear novel enough to hold as core contribution.
Reported performance gains are not surprising. Because GRPO explicitly rewards lower GED, observing lower GED after training is the expected outcome, and by itself this provides limited evidence of a novel research contribution. Although the evaluation includes packages unseen during training, gains on those packages are likewise not a significant finding, since the model is still being measured on the same metric it was trained to optimize. In addition, recent LLM decompilers work are less relying on single metrics to measure the output quality including R2I, re-compile, and re-exec over tests, while this paper doesn't measure on other metrics
Ref:
Zou, M., Cai, H., Wu, H., Basque, Z.L., Khan, A., Celik, Z.B., Tian, D., Bianchi, A., Wang, R., & Xu, D. (2025). D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning. ArXiv, abs/2506.10125.
Tan, H., Li, W., Tian, X., Wang, S., Liu, J., Li, J., & Zhang, Y. (2025). SK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin. ArXiv, abs/2509.22114.
Dramko, L., Le Goues, C., & Schwartz, E.J. (2025). Idioms: Neural Decompilation With Joint Code and Type Definition Prediction.
Questions for authors
It would be better to explain how this approach differs from D-lift and include it as a baseline, or discuss the difference/improvements.
Common evaluation includes the re-compile and re-exec as metrics to ensure that LLM will not change semantics of the decompiled program in drastic way pursuing readability or rewards. Is there any measure over this aspect?
There are established readability metrics such as R2I to measure the general decompiled output readability, providing this would help justify the approach’s effectiveness rather than single GED measurement.