- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GraphGuard, a neuro-symbolic framework for translating natural-language network-defense intents into device configurations. It combines a typed five-slot configuration graph, learned joint structural scoring, deterministic parameter solving, bounded repair, and backend compilation through an intermediate representation. Experiments cover overall synthesis quality, component ablations, compositional and linguistic generalization, runtime perturbations, search budgets, and backend portability. GraphGuard improves safe synthesis success over the evaluated retrieval, agentic, and symbolic baselines.
Critical strengths
-
The paper addresses a well-motivated problem in which natural-language interpretation must be combined with strict operational constraints. Its formulation captures dependencies among configuration choices, runtime resources, policies, and backend capabilities, providing a concrete setting for studying the interaction between learned semantic interpretation and symbolic reasoning.
-
The framework assigns clear responsibilities to its components. Learned scoring supports intent alignment and joint structural selection, while deterministic solving handles numerical feasibility. The typed configuration representation connects these stages and provides a consistent basis for checking dependencies, conflicts, resource limits, and backend realizability.
-
Verification, repair, and compilation are integrated into a coherent release procedure. Repairs are restricted to enumerated legal edits and followed by rechecking, while the intermediate representation supports explicit checks of values, units, scope, and lifecycle across backends. The paper also clearly states the conditions under which these checks provide guarantees.
-
The empirical evaluation is comparatively broad and goes beyond a single overall accuracy table. It includes agentic and symbolic baselines, extensive component ablations, distribution shifts, runtime counterfactuals, and budget analysis. Separating unsafe acceptance, over-refusal, structural correctness, and conditional executability helps reveal trade-offs that would be obscured by reporting success rate alone.
Critical weaknesses
-
The Symbolic-Only baseline provides limited evidence about the advantage over strong symbolic synthesis. Its compositional-OOD performance falls to zero because it uses exact-motif lookup, whereas a symbolic system could potentially recombine known prototypes under the same constraints. Since compositional generalization is a central claimed benefit, a comparison using the same candidates, solver, and verifier with deterministic structural ranking would more directly establish the contribution of learned joint scoring.
-
The additional value of LLM-guided repair is not convincingly established. On the initially invalid proposal pool, its recovery rate is 7.7%, compared with 7.1% for severity-greedy selection under the same attempt limit. This small conditional difference needs uncertainty and cost analysis, particularly because LLM selection introduces additional inference calls. The ablation supports the usefulness of bounded repair as a whole, but does not establish that its decision policy needs an LLM.
-
Some safety-related ablation results require a clearer causal explanation. Removing hard search constraints increases unsafe acceptance from 1.50% to 12.30%, although the intended final verifier retains the relevant checks. A downstream verifier should still reject violations of constraints it checks correctly. A stage-level breakdown is needed to explain whether these additional unsafe releases arise from different constraint coverage, semantic selection errors, or changes in the release path, and to distinguish the effects of pruning from verification.
-
The practical safety evidence depends heavily on the correctness and coverage of the frozen configuration graph and backend contracts. The experiments examine runtime and constraint changes, but do not establish robustness to missing dependencies, incorrect compatibility relations, or inaccurate resource bounds. Backend acceptance and round-trip consistency provide useful evidence of configuration preservation, while the reliability of the underlying knowledge remains insufficiently evaluated. This limits how broadly the reported safety results can be interpreted.
Questions for authors
-
How would a compositional symbolic baseline perform when given the same candidate prototypes, hard constraints, numerical solver, and final verifier, with deterministic ranking replacing learned structural scores? This would help assess the contribution of neural scoring beyond exact-motif lookup.
-
What explains the increase in unsafe acceptance when search-stage hard constraints are removed while the final verifier is retained? Please identify the violated conditions and the stage at which the additional unsafe configurations pass through the system.
-
Does LLM-guided repair improve final safe synthesis success over severity-greedy repair under comparable computational budgets? Please report the corresponding inference cost, latency, and uncertainty on the same invalid proposal pool.
-
What evidence establishes the accuracy and coverage of the constructed configuration graph? How sensitive are the results to a missing dependency, an incorrect relation, or an inaccurate numerical bound? Please also identify the actual graph and model configurations used to produce the reported results.
3 comments
Thank you so much for writing the response, I will retain my score as 3.
Thank You for Your Support and Constructive Review
Dear Reviewer JwqH,
Thank you very much for taking the time to read our response and revised manuscript, and for your positive evaluation and encouraging support of our work. Your constructive feedback has significantly helped us strengthen the clarity and rigor of the paper.
Best regards,
The Authors
Response to Reviewer JwqH
We thank Reviewer JwqH for the insightful review and for recognizing our problem formulation and the breadth of our empirical evaluation. We have incorporated your suggested analyses into the revised manuscript:
Compositional Symbolic Baseline (
Comp. Symbolic, Question 1):Comp. Symbolicin Table 1 (RQ1), Figure 3a (RQ3), and Appendix E.1 (Table 6). It uses the exact same candidate sets Kq,t, hard predicates Ωq (B=8), CP-SAT solver, and verifier Jver as GraphGuard, but replaces learned scoring EΘ with deterministic BM25 lexical similarity.Symbolic-Only) drops to 0.0%,Comp. Symbolicachieves 76.20% JEM and 58.86% SSS, proving our five-slot graph representation and hard constraints Ωq enable structural recombination. Crucially, GraphGuard's learned neural potentials EΘ provide an additional +15.31% JEM and +14.20% SSS gain on Compositional-OOD (and +10.05% JEM / +8.56% SSS on IID), showing the value of learned relational scoring.Stage-Level Breakdown of Search Pruning vs. Verification (Question 2):
w/o Search Pruningfromw/o Hard Constraintsin Table 2 (RQ2) and Appendix E.1:w/o Search Pruning(disabling Ωq during beam search while keeping the full downstream verifier Jver): Search pruning drops to 0.00% and verifier rejection jumps to 58.40%. Because the verifier still enforces relational rules, UAR remains statistically unchanged (1.65% vs. 1.50%), confirming that the verifier blocks relational violations. However, unpruned conflicting prefixes saturate the beam (B=8) and crowd out valid motifs, causing ORR to nearly triple (from 4.80% to 14.20%), JEM to drop by 9.25 points, and local P95 latency to double (4.610 ms).Cost, Latency, and Semantic Value of LLM-Guided vs. Severity-Greedy Repair (Question 3):
Severity-Greedyrestores constraint feasibility on 68.70% of cases, it greedily minimizes violation weight without intent awareness, achieving 51.20% conditional semantic SSS. In contrast,LLM-Guidedrepair achieves 68.40% conditional semantic SSS (+17.20% relative improvement, p<0.001) while requiring 85.81% fewer solver/verifier calls per invalid case (2.10 vs. 14.80 calls).Configuration Graph Accuracy and Corruption Sensitivity (Question 4):
We have provided the policy-permitted core algorithm-related code and experimental records in the supplementary package for your review. If our clarification and revision address your concerns, we hope you will carefully review the content of our paper again. We would be most grateful if you could consider raising the score.