- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes NSAI-FUZZ, a unified fuzzing framework for heterogeneous neuro-symbolic AI systems. It combines five guidance signals with two adaptive weighting methods, RCW and CSG, to improve failure discovery. Experiments across nine dataset–model combinations show consistent improvements over fixed-weight guidance.
Critical strengths
The paper addresses a relevant problem and provides a unified testing framework for heterogeneous neuro-symbolic systems. The evaluation covers multiple tasks and models, with ablation studies illustrating the varying importance of different guidance signals.
Critical weaknesses
Limited methodological novelty. The five guidance signals used in this paper are largely conventional metrics. Therefore, the main contribution appears to be a metric-fusion algorithm, while the proposed fusion mechanism seems relatively simple and relies on standard weighting and linear combination techniques, rather than introducing a fundamentally novel algorithm.
Questions for authors
What is the fundamental methodological novelty of RCW and CSG beyond combining established signals and weighting mechanisms?
We thank the reviewer for raising the question about methodological novelty. We agree that the individual signals are not intended to be novel in isolation. Our contribution is instead to address a problem specific to heterogeneous neuro-symbolic systems: different systems expose different intermediate interfaces, and the same signal can have substantially different failure-ranking reliability across dataset--model pairs and across search stages.
First, NSAI-Fuzz does not directly combine raw conventional metrics. It introduces an availability-aware, rank-normalized risk interface that maps heterogeneous neural and symbolic states into five comparable risk semantics. This allows the same search procedure to operate across systems such as ISED, DOLPHIN, NASR, and NeSyCoCo despite their different internal representations and unavailable signals.
Second, RCW is not an empirically chosen weighted sum. It estimates the failure-ranking reliability of each signal using AUROC and solves a KL-regularized allocation problem relative to a prior. This optimization has a closed-form solution, so the resulting weights explicitly reflect dataset--model-specific evidence while avoiding unstable allocations from limited calibration data.
Third, CSG addresses a different limitation of fixed fusion: signal usefulness can change during the search itself. It constructs a context from the current candidate pool and predicts availability-masked weights using a learned gate trained with a pairwise failure-versus-nonfailure ranking objective. Thus, RCW adapts at the dataset--model level, whereas CSG adapts at the search-round level.
Our ablations support that these adaptations are necessary rather than cosmetic. Fixed weighting outperforms Uniform in every dataset--model cell, while single-signal and leave-one-out results show substantial variation in which signals are useful. More importantly, both RCW and CSG outperform Fixed in all nine cells, with gains of up to 6.621 percentage points. Against the strongest matched neural-fuzzing baseline, the aggregate failure discovery rate increases from 6.06% to 16.17%/16.06% for RCW/CSG.
We will revise the paper to make this distinction clearer: the novelty is not the invention of five individual metrics, but the unified risk interface and reliability/context-adaptive guidance mechanisms that make heterogeneous NSAI systems fuzzable under a common framework.