- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces BLANKSPACE, a coding-agent workflow for reconstructing Swift, C, and Go repositories from stripped binaries. It combines extracted assembly and metadata with compilation, binary-similarity feedback, and automatic reversion. It also introduces BLANKSPACEBENCH, which evaluates 90 projects using their own tests and tests from downstream clients.
Critical strengths
- The benchmark addresses a useful evaluation gap. Whole-module replacement and downstream client tests check whether reconstructed code works within a project. The authors also check that downstream tests execute the target library.
- Figure 1 explains the motivation well through a concrete Swift example. Section 5.3 is also clear about the different rankings under static and functional metrics.
Critical weaknesses
- The results do not establish a consistent functional advantage for BLANKSPACE. On Swift, the framework improves static binary similarity, but upstream whole-module success falls from 3/38 to 2/38 and downstream success from 12/33 to 8/33. Function-level success is also slightly lower. C and Go show mixed or unchanged functional results. The conclusion acknowledges this limitation, so my concern is not inaccurate reporting. It is that the added framework has not demonstrated an advantage under the paper’s own behavioral criterion for successful reconstruction. The benchmark is useful, but this gap is central to my assessment of the method contribution.
- The ablation leaves the contribution of the feedback mechanisms unresolved. Removing scoring also removes assembly differences and automatic reversion. Table 2 omits the full-system baseline and reports no functional metrics. The results therefore cannot separate the effects of these components or establish whether any improves functional reconstruction. This matters because the feedback loop is a central part of the proposed method, and the overall comparison has not shown a consistent functional benefit.
Questions for authors
- Clarification of the functional evaluation: Why do the Swift functional metrics cover 38, 33, and 22 of the 50 projects? Were these subsets fixed before evaluating the reconstructions, and were all reconstruction build failures counted as failures within them? This would help determine whether the reported comparisons are affected by which projects could be evaluated.
- Clarification of the ablations: Is the full Codex + BLANKSPACE result in Table 6 the matched baseline for Table 2? If so, removing assembly differences gives higher instruction and BinDiff similarity. How should this result be interpreted, and do existing runs provide evidence separating the effects of scoring feedback and automatic reversion? This would affect my assessment of which design choices are supported.
3 comments
Questions on the Revised Claims and Main Contribution
Thank you for the additional experiments and clarifications. The revised ablations address much of my earlier concern about missing component comparisons. I also acknowledge the positive functional results in some settings, particularly for Go.
My main concern now is the interpretation of the ablations. Section 5.4 describes the configurations and summarizes static similarity, but does not explain the new functional results. With automatic reversion disabled, adding assembly diffs improves downstream whole-module success from 5/15 to 6/15; with reversion enabled, it decreases from 7/15 to 6/15. The full framework performs better on downstream function-level tests, but those results cover only four projects. These mixed outcomes do not establish a consistent benefit, but they may reveal something useful about how the components interact. How do the authors interpret these differences, and what motivates retaining both mechanisms in the final framework?
This also relates to my broader uncertainty about the paper's contribution. The expanded Swift evaluation still shows no whole-module advantage over plain Codex, despite improvements in static similarity and some function-level results elsewhere. I see value in both the reusable multi-language workflow and the repository-level benchmark, but it remains unclear whether the main contribution is improved decompilation performance or a new evaluation and understanding of its limitations. A clearer interpretation of the existing evidence and how it supports the intended contribution would help me reassess the paper.
Thanks for the feedback. We have just updated Table 1 and Table 2 with more functional correctness results. For Table 1, whole-module evaluation expanded from 38 to 42 upstream projects, and 33 to 47 downstream projects. The function-level evaluation now covers 30 upstream and 33 downstream projects. For the Table 2 ablations, the function-level results now cover 12 upstream and 14 downstream projects, extending the earlier 4 project results. We have also updated our observations from the Table 2 results in Section 5.4.
In Table 2 comparing "w/o Assembly Diffs and Automatic Reverts" with "w/o Automatic Reverts", upstream function-level success increases from 37.03% to 38.50%, while downstream function-level success decreases from 32.85% to 31.86%, and downstream whole-module success increases from 33.33% to 40%. Full BlankSpace achieves higher downstream function-level success than "w/o Assembly Diffs" at 37.26% vs. 30.86%, though downstream whole-module success is lower (40% vs. 46.67%).
The function-level success results suggest that assembly diffs can help reconstruct individual functions, even when they do not improve binary similarity or whole-module success. The expanded main Swift comparison in Table 1 also shows higher function-level success than plain Codex: 24.72% versus 22.52% upstream and 30.92% versus 24.66% downstream.
Our main contribution is a reusable workflow for repository-level decompilation across Swift, C, and Go, and BlankSpaceBench for evaluating the reconstructed code. The expanded results demonstrate improvements in binary similarity and individual function reconstruction but identify a current limitation that these improvements don’t consistently translate into higher whole-module success.
We thank the reviewer for their questions and suggestions. We have added 2 more ablation experiment modes to assess the effects of removing assembly diffs from BlankSpace in Table 2. For any revisions mentioned, we have made the revision font blue in the revised, uploaded pdf.
Why do the Swift functional metrics cover only 38, 33, and 22 of the 50 projects - At submission, Table 1 reported functional measurements for 38 projects for upstream whole-module evaluation, 33 with verified paired downstream whole-module results, and 22 for upstream single-function evaluation. The remaining measurements were unavailable because of reference test suite failures, inventory issues, or incomplete evaluation. Since submission, whole-module evaluation has expanded to 42 of the 50 Swift benchmark projects using upstream tests and 47 projects using downstream client tests.
Were the subsets fixed before you evaluated the reconstructions? - No, the subsets weren’t fixed before we evaluated the reconstructions.
Were build failures counted as failures? - Reconstruction build failures were counted as failures.
(NEW EXPERIMENTS) To test removing assembly diffs, we added two more ablation experiments on the same 20 Swift projects - 1) Scoring without assembly diffs or automatic reversion and 2) scoring with assembly diffs but without automatic reversion in Table 2. Without assembly diffs, instruction and BinDiff similarity are 46.00% and 58.71%, compared with 44.47% and 57.58% with assembly diffs. Assembly diffs provide localized diagnostics to guide edits, but these results show that such guidance does not consistently improve binary similarity.
Functional results show a different pattern. On the same 13 upstream and 15 downstream projects evaluated across all six ablation configurations, adding assembly diffs leaves upstream whole-module success unchanged at 15.38% and increases downstream success from 33.33% to 40.00%. These new ablations isolate the effect of assembly diffs with automatic reversion disabled.