acceptodds
Under review as a conference paper at ICLR 2027

S-ISP: Second-Order Semantic Peer Aggregation for LLM-Based Smart-Contract Vulnerability Detection

Abstract

Large language models can recognize semantic smart-contract vulnerabilities, yet repeated audits often share the same confident errors: ordinary voting measures the popularity of a finding, not whether the agreement is informative. We present Soft Inverse Surprising Popularity (S-ISP), a second-order semantic-peer aggregation principle for LLM auditing. Error-driven rule evolution constructs complementary auditors with distinct operational views. S-ISP compares their continuous scores with development-estimated peer behavior under inverse states, revealing support that is unexpected under correlated errors. Bilateral reliability weights both interaction endpoints, while a margin projection preserves the frozen first-order decision. Across real-contract, project-disjoint, and cross-taxonomy evaluations with open-weight LLM backbones, S-ISP consistently improves vulnerability discrimination and probability quality over prompt-based, multi-agent, and learned aggregation baselines while reducing deployment cost. A compact two-peer deployment also halves auditor calls. Component ablations attribute the principal accuracy and calibration gains to evolved-peer construction and inverse-peer calibration, respectively. All results follow a prespecified protocol with explicit separation of training, development, and test data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vulnerability Detection, LLM agents, probabilistic calibration, second-order aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pZcfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes S-ISP, a second-order semantic peer aggregation framework for LLM-based smart-contract vulnerability detection. To address the challenge that multiple LLM auditors may suffer from correlated errors and traditional voting struggles to distinguish effective consensus from redundant consensus, the authors first construct semantic auditors with distinct vulnerability assessment perspectives through error-driven rule evolution. They then estimate the inverse-state conditional dependencies among auditors using a development set to adjust the aggregated prediction probabilities accordingly. The proposed approach further incorporates bilateral reliability weighting, sparse auditor selection, and decision-preserving projection to improve prediction calibration while managing inference costs.

Critical strengths
  1. The research has clear motivation, and addresses an important limitation of multi-agent LLM auditing
  2. The combination of semantic peer evolution, inverse-state aggregation, and decision-preserving calibration provides a structured approach to improving vulnerability detection and probability estimation.
  3. The method achieves strong F1 and Brier scores across multiple datasets and vulnerability categories.
Critical weaknesses
  1. At the deployed two-peer operating point, the inner reliability weights in Eq. (4) cancel algebraically, leaving only the outer weighted sum. Moreover, all primary categories select uniform weights, and reentrancy uses zero correction strength. This raises questions about the practical contribution of the proposed bilateral weighting mechanism.
  2. Although S-ISP is motivated by continuous-score surprise, the inverse-state correction in Eq. (4) depends on test-time auditor outputs only through binary threshold indicators. Different confidence patterns with identical binary decisions therefore yield the same inverse-state correction term, making its distinction from simpler state-conditioned calibration less clear.
  3. The Brier-risk improvement results rely on favorable alignment between the correction and prediction residuals. Although the paper provides sufficient conditions for this alignment, the connection between these assumptions and the proposed semantic peer construction remains insufficiently established.
Questions for authors
  1. Given that the primary deployment uses two peers with uniform weights, what practical benefit does bilateral reliability weighting provide beyond the simpler outer-weighted formulation?
  2. How does the proposed inverse-state correction capture instance-specific semantic surprise when its test-time dependence is limited to binary peer decisions? What distinguishes it from simpler state-conditioned calibration methods?
  3. To what extent are the theoretical alignment assumptions supported by the error-driven semantic peer construction? Are there identifiable conditions under which the proposed inverse-state statistic may fail to provide useful calibration?
  4. Could the authors clarify the implementation and role of the source-structural access gate, as well as the main configuration differences among the analysis caches? How should readers interpret their results relative to the primary evaluation?
3 comments

Thank you for the detailed responses and additional experiments. I appreciate the clarification of the two-peer reduction, the new calibration baseline, and the distinctions among the analysis caches.

The responses address several of my concerns, but two points remain unclear. First, since the inverse-state correction preserves hard decisions, the reported F1 differences between outer-only and full bilateral weighting for three and four peers require further clarification regarding changes in support selection or outer weights. Second, under the primary two-peer uniform configuration, the examples (0.90, 0.10) and (0.55, 0.45) yield the same first-order probability and binary pattern, and thus the same corrected output. They therefore do not fully demonstrate instance-specific continuous surprise.

Overall, the rebuttal improves my understanding and strengthens the empirical evidence, but my concerns about the distinct contribution of the proposed mechanism remain. I will maintain my original score.

Thank you for the careful follow-up. The two observations help distinguish the architecture-level comparison from the matched calibration analysis, and they also allow us to explain the continuous role of the first-order anchor more directly.

1. Why F1 can differ across the three- and four-peer weighting rows

The decision-preserving guarantee is defined relative to a fixed first-order predictor. With support and outer weights fixed, projection retains that predictor's hard decision, so the matched effect of the second-order term is measured through probability quality, particularly Brier and ECE.

The exact-cardinality table compares complete architectures selected independently on development data under the same search space and tuning budget. At three peers, this selection includes the support and outer weights. At four peers, the support is necessarily all four auditors, while the outer weights are still fitted for each mode. Each row then projects around its own selected first-order predictor. Consequently, the F1 values describe the result of the complete development-selected architectures, while Brier and ECE describe their probability quality.

This gives the table two complementary interpretations. First, the exact two-peer equality verifies the algebraic cancellation of the inner weight. Second, for three or more peers, the table evaluates the full mode-specific architecture, including its development-fitted first-order component and second-order probability estimator. When support and outer weights are matched, inner weighting acts through the calibrated probability and is therefore evaluated by Brier/ECE at unchanged F1. The development selector ultimately retains the compact two-peer uniform model, which also provides the strongest operating point in the table.

2. State-conditioned signal and continuous anchor

The pair (0.90,0.10) and (0.55,0.45) illustrates an intentional invariance: both inputs have pattern (1,0) and uniform first-order probability 0.50, so the estimator maps them to the same corrected output. This behavior provides compact regularization of within-cell variation for the available development set.

A complementary comparison shows how continuous evidence affects the output: (0.90,0.30) and (0.70,0.40) share the same binary pattern (1,0) and the same vulnerable first-order decision, while their continuous first-order probabilities are 0.60 and 0.55. They access the same frozen raw inverse-state cell. For a raw margin correction c=0.04 in the non-saturated region, their corrected probabilities are 0.58 and 0.53. Near the boundary, projection adapts the effective correction to the available margin while retaining the first-order decision.

The mechanism therefore combines a state-conditioned second-order signal with an instance-specific continuous anchor. Continuous information enters through (i) the development expectation of soft support stored in each directed conditional cell, (ii) the current continuous first-order probability, and (iii) the current margin used by projection. The resulting estimator is both compact enough for the available development set and sensitive to continuous evidence whenever the first-order anchor or admissible margin differs.

3. Distinct contribution of the mechanism

The closest matched comparison directly measures this contribution using the same auditors, records, sparse support, projection, and development-only tuning budget. The additive verdict-conditioned calibrator obtains F1/Brier/ECE 0.933/0.091/0.041 with zero changed decisions, whereas S-ISP obtains 0.933/0.0876/0.029, also with zero changed decisions. In the separate correction knockout with observations, support, weights, and hard decisions fixed, the inverse-state correction reduces Brier from 0.0813 to 0.0704 while F1 remains 0.970.

These results support the distinct technical contribution: a directed state-conditioned estimator whose cells contain continuous semantic support, combined with a continuous first-order anchor and a projection that separates calibration from classification. Compared with a generic state-conditioned calibrator, this structured correction provides additional probability-quality gains while retaining the selected decisions.

We hope this explanation, together with the matched calibration control and the architecture-level interpretation of the cardinality table, makes the mechanism's distinct contribution clearer. And we would be glad to provide any further detail.

We would like to gently note that we have submitted a revised manuscript and provided additional clarifications and experimental evidence addressing the reviewers’ concerns. We would sincerely appreciate any further feedback and would be happy to provide additional details if helpful.

Reviewer_MdbgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes S-ISP for aggregating LLM assessments of smart contract vulnerabilities. It constructs complementary auditors through error-driven rule evolution, then uses inverse-state peer statistics to adjust confidence scores while preserving binary decisions. Development-set selection reduces the number of deployed auditors. Experiments on three smart contract corpora assess detection performance, probability quality, and computational cost, with ablations separating the effects of auditor construction and score correction.

Critical strengths
  1. The paper addresses a practical problem: LLM auditors can make correlated errors. Constructing complementary auditors and accounting for their dependence is well motivated.
  2. The framework separates semantic-peer construction from decision-preserving calibration and makes their respective contributions to classification performance and probability quality explicit.
  3. The evaluation includes matched ablations, learned aggregation baselines, standard calibration methods, and cross-corpus tests. These controls help assess individual components and performance across datasets.
Critical weaknesses
  1. The reported metrics and sample counts appear inconsistent with the stated evaluation protocol. Under standard positive-class F1, mean squared-error Brier scoring, and a common 0.5 threshold, SmartLLM's SmartBugs-v3 access-control result of F1 = 0.876 implies a Brier upper bound of approximately 0.416, even allowing for rounding. The reported Brier score is 0.581. Under the same assumptions, S-ISP's bad-randomness, timestamp, and unchecked-call results require at least 253 evaluation pairs combined, exceeding the 248 pairs reported for all categories in Table 14.
  2. The paper cites classical Surprisingly Popular voting but does not discuss Inverse Surprising Popularity (ISP), introduced by Ai et al. (ICML 2026). ISP already evaluates conditional answer statistics under counterfactual peer answers, a construction closely related to S-ISP's inverse-state mechanism.
  3. The existing calibration and stacking baselines do not fully isolate the benefit of inverse-state conditioning. Once its parameters are fixed, the raw correction is an additive function of the auditors' binary verdicts. A simpler verdict-conditioned calibrator with the same decision-preserving projection would provide a closer comparison.
  4. The presentation is difficult to follow. Dense notation and frequent references to appendices obscure the core algorithm. Results from different prediction caches and configurations are also hard to track, making it difficult to connect the experimental evidence to the paper's central claims.
Questions for authors
  1. How can the metric and sample-count discrepancies be reconciled? Please specify the F1 and Brier definitions, confirm whether both metrics use identical prediction records, and clarify whether 248 counts unique contracts or contract–vulnerability pairs.
  2. How does S-ISP differ from ISP's counterfactual conditional-statistics construction, and what theoretical or empirical evidence demonstrates the value of its modifications?
  3. Does S-ISP outperform an additive verdict-conditioned calibrator using the same auditors, development data, projection, and a comparable tuning budget?
Reviewer_VrMkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Smart contract vulnerability is an important problem and LLMs have been utilized to assist the smart contract vulnerability detection, showing it can perform equally well or slightly better than supervised algorithms.

This paper presents a second order semantic peer integration approach S-ISP for LLM-based smart contract vulnerability detection. TFrom the best of my understanding, it tries to measure and utilize the contrast between positive and negative state conditions to distinguish minority evidence from the agreement. However the paper is hard to read.

The paper provides a brief discussion (one paragraph) on existing smart contract vulnerability detection methods in the literature. However, the experimental evaluation in Table 1 did provide convincing discussion of the measurement results obtained and why the proposed approach is better and in what situations the proposed approach fails.

Critical strengths

(+) Smart contract vulnerability is an important problem and LLMs have been utilized to assist the smart contract vulnerability detection, showing it can perform equally well or slightly better than supervised algorithms.

(+) The paper provides Table 1 to compare the proposed method with some existing solution approaches.

(+) The paper provides a brief discussion (one paragraph) on existing smart contract vulnerability detection methods in the literature.

Critical weaknesses

(-) The paper did not provide sufficient comparison to the LLM-based SOTA solutions to the smart contract vulnerability detection in both the design methodology and the experimental evaluation.

(-) It is unclear why the paper claims the process of Aduit-Select-Compare-Calibrate is novel, since this multi-step process is used in most of the LLM-based the smart contract vulnerability detection methods, including those mentioned in the related work and introduction of the paper.

(-) Section 4 is hard to read and it is unclear what novelty this section brings to the smart contract vulnerability detection, and how this section enhances your Aduit-Select-Compare-Calibrate process.

(-) The paper lacks example walk-through on the Section 4 and the Aduit-Select-Compare-Calibrate process.

(-) The experimental evaluation in Table 1 did provide convincing discussion of the measurement results obtained.

(-) Table 1 should be discussed with illustration by examples on how and why the performance numbers are obtained and compared. and why the proposed approach is better and in what situations the proposed approach fails

Questions for authors

Please address those weakness points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.