- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper uses a static analysis to generate several graphs structures for vulnerability detection in Java (call graph, dataflow graph, etc). However, these graphs often have gaps due to features that are difficult to resolve statically, such as reflection. Sembridge uses an LLM to fill in those gaps so that they can discover more vulnerabilities via graph queries/vulnerability path search. They outperform all evaluated baselines.
Critical strengths
It's a solid neurosymbolic approach, intuitive, and outperforms the baselines.
Generally I think it's a pretty good paper. It identifies a problem (vulnerability detection), proposes a solution using ML, and evaluates it against existing baselines.
Critical weaknesses
I think there are two issues with this paper.
The first is novelty. Specifically, it is very similar to CaLLME: Call Graph Augmentation with Large Language Models for Javascript, COLM 2025, https://openreview.net/forum?id=xZi2rMUcAO, where the idea is to use an LLM to fill in gaps in static analysis that are very difficult to resolve statically. I feel the differences are all very superficial (CaLLME is just the call graph, Sembridge runs iteratively while CaLLME runs just once). The general idea of using an LLM to fill in the gaps of a static analysis I think has been done many times, so the novelty is kind of low.
The second is that it's unclear what the contribution of the paper is, although that's a general criticism of neurosymbolic approaches these days. It's a lot less clear that the SOTA is included, because I wouldn't be surprised at all if Mythos/a more powerful model would outperform all these approaches. I think the framing needs to be a bit clearer with that. Is the benefit of Sembridge that it allows smaller models to punch above their weight class, and therefore save tokens? I think at the very least it needs to be addressed in the limitations - it's not an accept-blocker for me.
Questions for authors
Mainly the weaknesses above.
-
What is the novelty/research contribution over CaLLME or other neurosymbolic approaches that use an LLM to address the blind spots of static analysis?
-
Is this a we-are-SOTA paper, or a token-saving/cost savings paper? If it's a we-are-SOTA paper, is there good reason to think that Sembridge will still have good improvements if the model is replaced with Mythos? This is more of a framing issue than anything else.