- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes S-ISP, a second-order semantic peer aggregation framework for LLM-based smart-contract vulnerability detection. To address the challenge that multiple LLM auditors may suffer from correlated errors and traditional voting struggles to distinguish effective consensus from redundant consensus, the authors first construct semantic auditors with distinct vulnerability assessment perspectives through error-driven rule evolution. They then estimate the inverse-state conditional dependencies among auditors using a development set to adjust the aggregated prediction probabilities accordingly. The proposed approach further incorporates bilateral reliability weighting, sparse auditor selection, and decision-preserving projection to improve prediction calibration while managing inference costs.
Critical strengths
- The research has clear motivation, and addresses an important limitation of multi-agent LLM auditing
- The combination of semantic peer evolution, inverse-state aggregation, and decision-preserving calibration provides a structured approach to improving vulnerability detection and probability estimation.
- The method achieves strong F1 and Brier scores across multiple datasets and vulnerability categories.
Critical weaknesses
- At the deployed two-peer operating point, the inner reliability weights in Eq. (4) cancel algebraically, leaving only the outer weighted sum. Moreover, all primary categories select uniform weights, and reentrancy uses zero correction strength. This raises questions about the practical contribution of the proposed bilateral weighting mechanism.
- Although S-ISP is motivated by continuous-score surprise, the inverse-state correction in Eq. (4) depends on test-time auditor outputs only through binary threshold indicators. Different confidence patterns with identical binary decisions therefore yield the same inverse-state correction term, making its distinction from simpler state-conditioned calibration less clear.
- The Brier-risk improvement results rely on favorable alignment between the correction and prediction residuals. Although the paper provides sufficient conditions for this alignment, the connection between these assumptions and the proposed semantic peer construction remains insufficiently established.
Questions for authors
- Given that the primary deployment uses two peers with uniform weights, what practical benefit does bilateral reliability weighting provide beyond the simpler outer-weighted formulation?
- How does the proposed inverse-state correction capture instance-specific semantic surprise when its test-time dependence is limited to binary peer decisions? What distinguishes it from simpler state-conditioned calibration methods?
- To what extent are the theoretical alignment assumptions supported by the error-driven semantic peer construction? Are there identifiable conditions under which the proposed inverse-state statistic may fail to provide useful calibration?
- Could the authors clarify the implementation and role of the source-structural access gate, as well as the main configuration differences among the analysis caches? How should readers interpret their results relative to the primary evaluation?
3 comments
Thank you for the careful follow-up. The two observations help distinguish the architecture-level comparison from the matched calibration analysis, and they also allow us to explain the continuous role of the first-order anchor more directly.
1. Why F1 can differ across the three- and four-peer weighting rows
The decision-preserving guarantee is defined relative to a fixed first-order predictor. With support and outer weights fixed, projection retains that predictor's hard decision, so the matched effect of the second-order term is measured through probability quality, particularly Brier and ECE.
The exact-cardinality table compares complete architectures selected independently on development data under the same search space and tuning budget. At three peers, this selection includes the support and outer weights. At four peers, the support is necessarily all four auditors, while the outer weights are still fitted for each mode. Each row then projects around its own selected first-order predictor. Consequently, the F1 values describe the result of the complete development-selected architectures, while Brier and ECE describe their probability quality.
This gives the table two complementary interpretations. First, the exact two-peer equality verifies the algebraic cancellation of the inner weight. Second, for three or more peers, the table evaluates the full mode-specific architecture, including its development-fitted first-order component and second-order probability estimator. When support and outer weights are matched, inner weighting acts through the calibrated probability and is therefore evaluated by Brier/ECE at unchanged F1. The development selector ultimately retains the compact two-peer uniform model, which also provides the strongest operating point in the table.
2. State-conditioned signal and continuous anchor
The pair (0.90,0.10) and (0.55,0.45) illustrates an intentional invariance: both inputs have pattern (1,0) and uniform first-order probability 0.50, so the estimator maps them to the same corrected output. This behavior provides compact regularization of within-cell variation for the available development set.
A complementary comparison shows how continuous evidence affects the output: (0.90,0.30) and (0.70,0.40) share the same binary pattern (1,0) and the same vulnerable first-order decision, while their continuous first-order probabilities are 0.60 and 0.55. They access the same frozen raw inverse-state cell. For a raw margin correction c=0.04 in the non-saturated region, their corrected probabilities are 0.58 and 0.53. Near the boundary, projection adapts the effective correction to the available margin while retaining the first-order decision.
The mechanism therefore combines a state-conditioned second-order signal with an instance-specific continuous anchor. Continuous information enters through (i) the development expectation of soft support stored in each directed conditional cell, (ii) the current continuous first-order probability, and (iii) the current margin used by projection. The resulting estimator is both compact enough for the available development set and sensitive to continuous evidence whenever the first-order anchor or admissible margin differs.
3. Distinct contribution of the mechanism
The closest matched comparison directly measures this contribution using the same auditors, records, sparse support, projection, and development-only tuning budget. The additive verdict-conditioned calibrator obtains F1/Brier/ECE 0.933/0.091/0.041 with zero changed decisions, whereas S-ISP obtains 0.933/0.0876/0.029, also with zero changed decisions. In the separate correction knockout with observations, support, weights, and hard decisions fixed, the inverse-state correction reduces Brier from 0.0813 to 0.0704 while F1 remains 0.970.
These results support the distinct technical contribution: a directed state-conditioned estimator whose cells contain continuous semantic support, combined with a continuous first-order anchor and a projection that separates calibration from classification. Compared with a generic state-conditioned calibrator, this structured correction provides additional probability-quality gains while retaining the selected decisions.
We hope this explanation, together with the matched calibration control and the architecture-level interpretation of the cardinality table, makes the mechanism's distinct contribution clearer. And we would be glad to provide any further detail.
We would like to gently note that we have submitted a revised manuscript and provided additional clarifications and experimental evidence addressing the reviewers’ concerns. We would sincerely appreciate any further feedback and would be happy to provide additional details if helpful.
Thank you for the detailed responses and additional experiments. I appreciate the clarification of the two-peer reduction, the new calibration baseline, and the distinctions among the analysis caches.
The responses address several of my concerns, but two points remain unclear. First, since the inverse-state correction preserves hard decisions, the reported F1 differences between outer-only and full bilateral weighting for three and four peers require further clarification regarding changes in support selection or outer weights. Second, under the primary two-peer uniform configuration, the examples (0.90, 0.10) and (0.55, 0.45) yield the same first-order probability and binary pattern, and thus the same corrected output. They therefore do not fully demonstrate instance-specific continuous surprise.
Overall, the rebuttal improves my understanding and strengthens the empirical evidence, but my concerns about the distinct contribution of the proposed mechanism remain. I will maintain my original score.