acceptodds
Under review as a conference paper at ICLR 2027

One Fuzzer, Many Reasoners: Adaptive Neuro-Symbolic Fuzzing with Reliability-Calibrated and Context-Gated Guidance

Abstract

Neural-symbolic AI (NSAI) combines learned perception with programs, constraints, and compositional reasoning, but remains difficult to test because small semantics-preserving changes can alter grounded symbols and trigger discontinuous reasoning failures. Existing neural fuzzers are typically tied to architecture-specific representations and lack a common interface for heterogeneous NSAI systems. To the best of our knowledge, we introduce the first unified fuzzing framework specifically designed for heterogeneous NSAI. NSAI-FUZZ separates task-specific semantics-preserving mutation from architecture-neutral search and represents each candidate using five rank-normalized signals: boundary pressure, grounding uncertainty, representation drift, constraint violation, and archive novelty. We further propose Reliability-Calibrated Weighting (RCW), which learns dataset–model-level weights from signal reliability, and Contextual Signal Gating (CSG), which adapts weights to the current candidate pool. We evaluate ISED, DOLPHIN, NASR, and NeSyCoCo on four reasoning tasks, covering nine compatible dataset–model cells and 1,350 on-policy runs.Fixed five-signal guidance outperforms Random and Uniform in every cell, by up to 19.115 pp and 3.280 pp, respectively, while single-signal and leave-one-out analyses reveal strong model–task dependence in signal importance. Motivated by this heterogeneity, both RCW and CSG improve the mean failure discovery rate over Fixed in all cells, with gains of up to 6.621 pp. Wall-clock evaluation shows that the adaptive methods retain essentially Fixed-like cost while achieving 5.10× and 5.08× the failure throughput of Random. These results establish fuzzing as a practical testing methodology for heterogeneous NSAI systems and show that effective guidance should adapt to the neural–symbolic interface.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neural-Symbolic AI, Fuzz Testing, Reliability Testing, Adaptive Guidance, Semantics-Preserving Mutation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SQUDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NSAI-FUZZ, a unified fuzzing framework for heterogeneous neuro-symbolic AI systems. It combines five guidance signals with two adaptive weighting methods, RCW and CSG, to improve failure discovery. Experiments across nine dataset–model combinations show consistent improvements over fixed-weight guidance.

Critical strengths

The paper addresses a relevant problem and provides a unified testing framework for heterogeneous neuro-symbolic systems. The evaluation covers multiple tasks and models, with ablation studies illustrating the varying importance of different guidance signals.

Critical weaknesses

Limited methodological novelty. The five guidance signals used in this paper are largely conventional metrics. Therefore, the main contribution appears to be a metric-fusion algorithm, while the proposed fusion mechanism seems relatively simple and relies on standard weighting and linear combination techniques, rather than introducing a fundamentally novel algorithm.

Questions for authors

What is the fundamental methodological novelty of RCW and CSG beyond combining established signals and weighting mechanisms?

1 comment

We thank the reviewer for raising the question about methodological novelty. We agree that the individual signals are not intended to be novel in isolation. Our contribution is instead to address a problem specific to heterogeneous neuro-symbolic systems: different systems expose different intermediate interfaces, and the same signal can have substantially different failure-ranking reliability across dataset--model pairs and across search stages.

First, NSAI-Fuzz does not directly combine raw conventional metrics. It introduces an availability-aware, rank-normalized risk interface that maps heterogeneous neural and symbolic states into five comparable risk semantics. This allows the same search procedure to operate across systems such as ISED, DOLPHIN, NASR, and NeSyCoCo despite their different internal representations and unavailable signals.

Second, RCW is not an empirically chosen weighted sum. It estimates the failure-ranking reliability of each signal using AUROC and solves a KL-regularized allocation problem relative to a prior. This optimization has a closed-form solution, so the resulting weights explicitly reflect dataset--model-specific evidence while avoiding unstable allocations from limited calibration data.

Third, CSG addresses a different limitation of fixed fusion: signal usefulness can change during the search itself. It constructs a context from the current candidate pool and predicts availability-masked weights using a learned gate trained with a pairwise failure-versus-nonfailure ranking objective. Thus, RCW adapts at the dataset--model level, whereas CSG adapts at the search-round level.

Our ablations support that these adaptations are necessary rather than cosmetic. Fixed weighting outperforms Uniform in every dataset--model cell, while single-signal and leave-one-out results show substantial variation in which signals are useful. More importantly, both RCW and CSG outperform Fixed in all nine cells, with gains of up to 6.621 percentage points. Against the strongest matched neural-fuzzing baseline, the aggregate failure discovery rate increases from 6.06% to 16.17%/16.06% for RCW/CSG.

We will revise the paper to make this distinction clearer: the novelty is not the invention of five individual metrics, but the unified risk interface and reliability/context-adaptive guidance mechanisms that make heterogeneous NSAI systems fuzzable under a common framework.

Reviewer_tq2xOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents NSAI-FUZZ, a framework for testing heterogeneous neural-symbolic systems. Task-specific mutators produce inputs intended to preserve the correct answer. Model adapters extract five signals covering decision boundaries, grounding uncertainty, representation drift, constraint violations, and archive novelty. The framework ranks and combines these signals to select parents for further mutation. Reliability-Calibrated Weighting (RCW) learns weights for each dataset and model combination, while Contextual Signal Gating (CSG) adjusts weights across search rounds. Experiments cover nine compatible combinations and report higher mean candidate failure rates for both adaptive policies than for the predefined Fixed policy.

Critical strengths
  1. Separating semantic validity from search guidance gives the framework a clear structure. Adapters and explicit masking of unavailable signals provide a practical interface for systems with different internal representations.
  2. The evaluation controls mutation families, beam width, query budgets, and paired random seeds. Running each policy from scratch appropriately accounts for the different candidate pools induced by its search decisions.
  3. The single-signal and leave-one-out experiments provide useful diagnostics. The timing analysis also distinguishes sustained failure throughput from time to the first failure and acknowledges that guidance increases the latter.
Critical weaknesses
  1. The need for a dedicated neural-symbolic testing method is not established: Sections 1-3 motivate the framework through interface heterogeneity and discontinuous reasoning failures. However, the tested mutations largely affect visual appearance, and the paper provides little analysis separating ordinary perception failures from failures caused by the interaction between perception and symbolic execution. It remains unclear which failure mechanisms require the proposed guidance and which existing testing methods would miss them. The formal properties of rank fusion, masking, and weighting do not establish this connection. A concrete failure analysis is needed to support the central claim that neural-symbolic structure calls for this particular design.

  2. The baselines do not isolate the contribution of adaptive weighting: The experiments compare policies within NSAI-FUZZ but do not directly evaluate adapted versions of existing neural testing methods. RCW and CSG use labeled calibration candidates, whereas Fixed uses a predefined weight vector. There is no comparison with fixed weights optimized using the same calibration data and budget. Table 2 also shows that boundary guidance is the strongest individual signal in seven of nine settings. On HWF-7/DOLPHIN, it achieves 22.355%, compared with 18.098% for Fixed and 24.719% for RCW (Tables 1-2). These results make stronger comparisons essential. For CSG, a control with constant context or fixed output weights is needed to identify the benefit of changing weights during search.

  3. The primary metric leaves the practical value of discovered failures unresolved: Equation 21 measures the fraction of evaluated candidates that fail, and Section 4.5 allows search to continue after a failure. This objective can reward repeated exploration near an already discovered failure. A higher rate therefore does not establish that the method exposes weaknesses on more original inputs or discovers a wider range of failure mechanisms. The paper reports little evidence on either point. Seed-level success rates and an analysis of failure diversity would help establish the testing value of the additional samples. The current results primarily support increased failure yield.

  4. Implementation details and uncertainty reporting are insufficient for verification: Section 4.1 and Appendix B describe grounding uncertainty and constraint violation through adapter-dependent alternatives, without fully specifying their computation and aggregation for each setting. Exact representation choices and mutation parameters also need clarification. Preserving a latent scene or expression label does not by itself verify that a transformed image remains readable and semantically unambiguous. Section 5 does not clearly establish separation between calibration, screening, and final evaluation at the original-input level. In addition, Tables 1 and 6 give means over ten paired seeds without confidence intervals or paired uncertainty analysis. The 0.072 percentage-point RCW gain on ReaSCAN illustrates why positive mean differences alone are insufficient to support reliable improvement in every setting.

Questions for authors
  1. Which concrete neural-symbolic failure can the proposed signals uncover that representative existing neural testing methods fail to discover under the same mutation space, model access, and budget? Please provide a comparison and a failure trace.
  2. How do RCW and CSG compare with fixed weights optimized on the same calibration data? Please quantify their gains over boundary-only guidance with paired uncertainty estimates. For CSG, what remains of the gain when its context is held constant?
  3. Do the improvements persist when measured by the fraction of original inputs with at least one discovered failure? How diverse are the additional failures, and how are closely related cases grouped?
  4. What are the exact formulas and aggregation rules for G and C, the representations used for R and N, and the mutation bounds and sampling rules? How is semantic validity checked beyond retaining the original label?
  5. Are calibration, screening, and final evaluation separated by original input and mutation ancestry? Does the reported calibration time include generating the calibration traces, and how many model executions does collection require?
  6. Can the authors report confidence intervals for the paired differences across seeds and identify which adaptive gains remain supported after accounting for variability?
2 comments

We thank the reviewer for the detailed comments. We have substantially strengthened the evaluation with matched neural-fuzzing baselines, failure deduplication/diversity analysis, and a clearer characterization of RCW and CSG.

Q1. Comparison with existing neural fuzzers.
We now compare against DeepHunter, TensorFuzz, DLFuzz, NSGen-transfer, and CriticalFuzz under the same initially correct inputs, semantics-preserving mutations, validity checks, checkpoints, beam width, hardware, and 256-query budget.

Dataset/ModelDeepHTensorFDLFuzzNSGen-TCritFRCWCSG
SoC/ISED.181.911.31.39.1413.83213.526
SoC/DOL.1.351.601.50.591.3314.91415.557
Sudoku/ISED4.695.725.143.954.4916.06216.629
Sudoku/DOL.000005.7645.740
Sudoku/NASR9.969.0810.3110.029.5923.59823.740
HWF/ISED3.307.238.12.802.939.6809.934
HWF/DOL.3.679.3010.2012.323.6724.71923.123
HWF/NASR4.7714.8810.0018.095.9626.65025.633
ReaSCAN.744.843.892.255.5710.28310.664

TensorFuzz is the strongest neural-fuzzing baseline in aggregate at 6.06%, while RCW/CSG achieve 16.17%/16.06%, i.e., 2.67×/2.65× the TensorFuzz rate. On Sudoku/DOLPHIN, all five neural-fuzzing baselines find no failures, while RCW/CSG achieve 5.764%/5.740%.

Q2. Adaptive weighting vs. fixed guidance.
RCW is itself the calibrated fixed-weight control: it learns one weight vector per dataset-model cell from random-frontier calibration data and keeps it fixed during search. CSG additionally changes weights according to the current candidate-pool context. RCW wins four cells and CSG five, so we do not claim that CSG universally dominates RCW.

Boundary is the strongest one-hot signal in seven of nine cells, but it is not consistently optimal. For example, on HWF/DOLPHIN, Boundary reaches 22.355%, Fixed 18.098%, and RCW 24.719%. Grounding is strongest on SoC/DOLPHIN and Drift on HWF/ISED, while leave-one-out results show a different pattern. Both RCW and CSG outperform Fixed in all nine cells, with gains up to 6.621 percentage points.

Q3. Do the gains remain after grouping duplicate or similar failures?

Yes. We added exact deduplication and activation-similarity clustering across all nine settings.

PolicyRawExact uniqueSimilarity clusters
DeepHunter14,6705,3005,450
TensorFuzz27,9405,6106,210
DLFuzz25,8405,3905,850
NSGen-transfer24,7805,3305,740
CriticalFuzz17,2405,3705,620
RCW74,50610,71911,508
CSG74,00311,31112,058

Relative to TensorFuzz, RCW and CSG retain 1.85× and 1.94× as many similarity-clustered failure behaviors. The ordering is also stable for cosine thresholds from 0.95 to 0.99: the strongest neural baseline retains 5,910--6,590 clusters, compared with 10,982--12,021 for RCW and 11,659--12,638 for CSG.

Therefore, the improvement is not explained only by repeatedly sampling near-duplicate failures. We nevertheless agree that candidate-level failure rate and failure diversity measure different aspects of testing effectiveness. The current study establishes higher failure yield and higher observed failure diversity; it does not yet establish causal independence of the underlying failure mechanisms.

Q4. Signal definitions and semantic validity.
We expanded the method description. Grounding uncertainty uses normalized symbol entropy, prototype distance, or execution uncertainty depending on the exposed state. Constraint violation measures normalized symbolic inconsistency and is masked when unavailable. Representation drift and novelty use adapter-selected intermediate states. Semantic validity is task-specific: SoC preserves scene/question/answer, Sudoku preserves the unique solution, HWF preserves the expression, and ReaSCAN preserves scene graph/command/referent.

Q5. Calibration separation.
Calibration uses random-frontier traces only. RCW/CSG are calibrated from these pools, after which every reported policy is rerun from scratch on fresh on-policy trajectories. Up to 2,000 calibration candidates are retained per cell. We do not claim zero-shot transfer to unseen dataset-model settings; cross-setting calibration remains future work.

Q6. Uncertainty.
All values are means over the same ten paired seeds. We agree that paired confidence intervals are needed before interpreting very small cell-level differences as statistically significant. We therefore restrict our claim to observed mean improvements and emphasize the larger aggregate gaps: 6.06% for TensorFuzz versus 16.17%/16.06% for RCW/CSG, together with the substantially larger deduplicated failure sets.

Reviewer_tq2xOpenReview ↗

Response to Rebuttal

I thank the authors for the additional baseline comparisons and failure-diversity analysis. These additions strengthen the evaluation, and I acknowledge the clarification that RCW uses calibrated weights that remain fixed during search.

However, several central concerns remain insufficiently addressed. The response still lacks concrete failure traces establishing how the proposed guidance exposes failures arising from interactions between perception and symbolic execution. The benefit of context-dependent gating also remains insufficiently isolated without the requested constant-context ablation and paired uncertainty estimates. Furthermore, rerunning policies on fresh trajectories does not establish separation between calibration and evaluation at the original-input and mutation-ancestry levels. The semantic-validity checks and deduplication/clustering procedures require clearer operational definitions, and the fraction of original inputs with at least one discovered failure remains unreported.

While the reported gains are encouraging, these gaps limit my confidence in the attribution and practical testing value of the improvements. I therefore maintain my original score of 1.

Reviewer_oDwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NSAI-FUZZ, a unified fuzzing framework for heterogeneous neural-symbolic systems. It separates task-specific semantic mutations from a common search procedure guided by five risk signals. The authors introduce Reliability-Calibrated Weighting (RCW) and Contextual Signal Gating (CSG) to adapt signal weights. Experiments across four neural-symbolic model families and nine dataset-model settings show improvements in failure discovery rates over random selection and fixed weighting.

Critical strengths
  1. Separating semantics-preserving mutations from model-specific guidance provides a reasonable approach to testing heterogeneous neural-symbolic systems.

  2. The evaluation covers several reasoning architectures and tasks.

Critical weaknesses
  1. RCW and CSG are calibrated on random-frontier data for each dataset-model setting. Their performance under unseen settings or substantially different search distributions remains untested.

  2. The paper does not directly compare against established neural network fuzzing methods such as DeepHunter, TensorFuzz, or NSGen. Most experiments only compare variants of the proposed framework.

  3. The primary metric counts failing candidates rather than distinct failure patterns.

  4. Boundary pressure is the strongest individual signal in seven of nine settings. Some adaptive improvements over Fixed are marginal, and CSG does not consistently outperform RCW.

  5. The paper does not distinguish errors originating from neural perception, symbol grounding, or symbolic reasoning.

Questions for authors

Q1. Why were DeepHunter [1], TensorFuzz [2], and NSGen [3] not included as baselines? How does NSAI-FUZZ compare with them under matched mutation and query budgets?

Q2. Do the improvements remain after grouping duplicate or highly similar failures?

[1] Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., ... & See, S. (2019, July). Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis (pp. 146-157).

[2] Odena, A., Olsson, C., Andersen, D., & Goodfellow, I. (2019, May). Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. In International conference on machine learning (pp. 4901-4911). PMLR.

[3] Huang, L., Sun, W., Yan, M., Liu, Z., Lei, Y., & Lo, D. (2024). Neuron semantic-guided test generation for deep neural networks fuzzing. ACM Transactions on Software Engineering and Methodology, 34(1), 1-38.

2 comments

We thank the reviewer for the detailed feedback. We have added substantial new experiments and analyses that directly address the concerns.

Q1: Comparison with established neural fuzzers. We agree that this comparison is important. We have now added five matched neural-fuzzing baselines: DeepHunter, TensorFuzz, DLFuzz, NSGen-transfer, and CriticalFuzz. Because these methods were originally designed for conventional neural networks rather than heterogeneous neural-symbolic pipelines, we transfer their feedback principles to the same candidate stream while keeping all other factors fixed. Specifically, all methods use the same initially correct inputs, semantics-preserving mutation operators, semantic validity checks, model checkpoints, beam width, hardware, and 256-query budget.

Across the nine dataset-model settings, TensorFuzz is the strongest neural-fuzzing baseline in aggregate, with a 6.06% failure discovery rate. In comparison, RCW and CSG achieve 16.17% and 16.06%, corresponding to 2.67× and 2.65× the TensorFuzz rate. The advantage is also visible at the individual-setting level. For example, on Visual Sudoku/NASR, the strongest neural baseline reaches 10.31%, while RCW/CSG reach 23.598%/23.740%; on HWF-7/DOLPHIN, NSGen-transfer reaches 12.32%, compared with 24.719%/23.123% for RCW/CSG. On Visual Sudoku/DOLPHIN, all five neural-fuzzing baselines find no failures, whereas RCW/CSG still achieve 5.764%/5.740%.

Q2: Duplicate and highly similar failures. We have also added both exact deduplication and activation-similarity clustering. Exact deduplication retains one failure per clean-seed/erroneous-output pair, while similarity clustering further separates failures according to their activation representations. TensorFuzz retains 5,610 exact-unique failures and 6,210 similarity clusters. RCW retains 10,719 exact-unique failures and 11,508 similarity clusters, while CSG retains 11,311 and 12,058, respectively. Thus, RCW/CSG retain 1.85×/1.94× as many similarity-clustered failure behaviors as TensorFuzz. This ordering is robust to cosine thresholds from 0.95 to 0.99, showing that the gain is not explained by repeatedly rediscovering near-duplicate failures.

Boundary dominance and adaptive gains.
We agree that Boundary is the strongest isolated signal in seven of nine settings. However, the ablation results also show that isolated strength does not determine the best multi-signal allocation. Grounding is strongest on Sort-of-CLEVR/DOLPHIN and Drift on HWF-7/ISED, while leave-one-out results show a different pattern again. More importantly, both RCW and CSG outperform Fixed in all nine settings, with improvements of up to 6.621 percentage points. Thus, Boundary is often important, but no single fixed weighting is consistently optimal.

We also do not claim that CSG universally dominates RCW. RCW achieves the higher rate in four settings and CSG in five. Their purpose is complementary: RCW adapts to dataset-model-level signal reliability, whereas CSG adapts to candidate-pool context during search.

Overall, the added experiments now show that the improvement of NSAI-Fuzz persists against established neural-fuzzing baselines, under matched mutation and query budgets, and after both exact and similarity-based deduplication. We believe these additions substantially strengthen the empirical support for the proposed adaptive neural-symbolic guidance framework.

We thank the reviewer for these questions. We have added the requested comparisons and deduplication analyses.

Q1. Comparison with established neural fuzzers.

We now compare against DeepHunter, TensorFuzz, DLFuzz, NSGen-transfer, and CriticalFuzz under the same initially correct inputs, semantics-preserving mutations, validity checks, model checkpoints, beam width, hardware, and 256-query budget.

DatasetModelDeepHunterTensorFuzzDLFuzzNSGen-TCriticalFuzzRCWCSG
Sort-of-CLEVRISED0.181.911.310.390.1413.83213.526
Sort-of-CLEVRDOLPHIN1.351.601.500.591.3314.91415.557
Visual SudokuISED4.695.725.143.954.4916.06216.629
Visual SudokuDOLPHIN0.000.000.000.000.005.7645.740
Visual SudokuNASR9.969.0810.3110.029.5923.59823.740
HWF-7ISED3.307.238.120.802.939.6809.934
HWF-7DOLPHIN3.679.3010.2012.323.6724.71923.123
HWF-7NASR4.7714.8810.0018.095.9626.65025.633
ReaSCANNeSyCoCo0.744.843.892.255.5710.28310.664

TensorFuzz is the strongest neural-fuzzing baseline in aggregate at 6.06% failure discovery, while RCW and CSG achieve 16.17% and 16.06%, corresponding to 2.67× and 2.65× the TensorFuzz rate.

Q2. Deduplication and highly similar failures.

We additionally report exact deduplication and activation-similarity clustering across all nine settings.

PolicyRawExact uniqueSimilarity clusters
DeepHunter14,6705,3005,450
TensorFuzz27,9405,6106,210
DLFuzz25,8405,3905,850
NSGen-transfer24,7805,3305,740
CriticalFuzz17,2405,3705,620
RCW74,50610,71911,508
CSG74,00311,31112,058

Thus, compared with TensorFuzz, RCW and CSG retain 1.85× and 1.94× as many similarity-clustered failure behaviors. This ordering is also robust to cosine thresholds from 0.95 to 0.99, indicating that the improvement is not explained by repeated near-duplicate failures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.