- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TrustMe, an LLM-driven benchmark designed to evaluate code auditing and hardware Trojan detection in realistic hardware design contexts. The benchmark comprises 816 evaluation tasks with approximately 200,000 lines of code. It is structured into four subsets: Hardware Security Understanding, Atomic Capability Verification, Module-Level Auditing, and System-Level Auditing. The paper further employs a self-evolving agent-human collaborative workflow to generate stealthy, triggerable Trojans, validated by commercial tools and simulation. The paper evaluates 6 LLMs across 5 agent frameworks, observing that the best configuration only achieves a 37.5% pass rate. Thus, the paper proposes FindHT, a lightweight enhancement framework that improves the average pass rate by up to 19.83%.
Critical strengths
- As LLMs become pervasive in RTL generation, their ability to audit and detect hardware Trojans is a critical security bottleneck. TrustMe fills a significant gap in existing benchmarks.
- This work builds an self-evolving agent-human collaborative pipeline and contains 1560 functional Trojan programs, which effectively alleviates the issue of data scarcity in this research direction.
Critical weaknesses
-
Some data in the benchmark comes from TrustHub and textbooks. The paper mentioned that these data may be included in model pre-training data and affects evaluation results. Therefore, this benchmark cannot avoild the issue of data contaminating.
-
The paper lacks budget allocation experiments, component ablation studies, and cross-combination generalization analysis. It is still unclear whether the improvement of 12.6% comes from structured reasoning or from more tokens. It can be observed that the proposed method has achieved a certain performance improvement by incurring a higher cost. Whether this performance improvement is worthwhile still requires further consideration.
-
There seems no comparison with traditional Trojan detection tools, static analysis, or commercial EDA security tools. Without the comparison results, it is difficult to determine the significance of the proposed method.
Questions for authors
- Is there any comparison of FindHT with the baselines under the same token budget, the same time budget, and the same number of iterations?
- Why is the full-score rate of the M subset always 0? Does this result indicate that the scoring mechanism might be too strict, or that the task definition is unclear?
- Does the paper consider trying to fix this issue by changing the format, etc.?
- Is there any comparison with traditional Trojan detection tools, static analysis, or commercial EDA security tools.
We sincerely thank the reviewer for your time, careful reading, and kind recognition that the benchmark addresses a worthwhile gap and provides helpful experimental detail. Below is our point-to-point rebuttal.
A1
We have clarified the metric definitions and aggregation rules in Section 4, Appendix E.2, and the revised manuscript. Of the 816 items, 14 vision-only items are marked
unsupported_visionand excluded from the execution denominators for non-vision agentic runs, leaving 802 runnable items (225 inM). For each configuration, we run five independent trials per item.Pass@5indicates whether at least one of the five runs succeeds on the task, whereasPass ratein Table 1 is 1 minus the overall execution failure rate (including agent, LLM, or unrecovered infrastructure failures).Full-score rateis the fraction of scored items that achieve 100% of the rubric points.Average scoreis the mean rubric-normalized score over scored items with non-zero scores, excluding zero-score and execution-failed items. In Table 10 (deepseek-v4-pro + codexonM, 225 items), all 225 items were run, completed, and scored, with 5.8% (13 items) achieving full score. The 94.2% figure in theFail (%)column of the per-dimension table reflects items that did not achieve full score (100% − 5.8% = 94.2%). We have clarified this in the revised text and table captions to avoid conflating it with execution failure. Forbook_true_false(129 scored items), 89.1% reflects the rubric score across scored items as computed by our evaluation script, and we have explicitly documented the exact per-item and five-run aggregation formulas in Appendix E.2.A2
In our evaluation pipeline,
glm-5.2serves as the judge model for rubric scoring, as specified in Appendix E.2 and Appendix F. We have corrected the conflicting mention in Section 4 in the revision. Deterministic checks handle closed-choice and structured fields, while the judge model evaluates rubric criteria such as free-text descriptions. On a sampled subset audited by human hardware security experts, judge scores align closely with human rubric annotations on structured fields and key security points, and we have added a concise agreement summary in the revision.A3
Every implanted Trojan in our benchmark is constructed and checked through our verification workflow (Appendix C), including syntax/lint checking, simulation against the design testbench, and synthesis checks. These steps ensure design validity and confirm triggerability and payload behavior, rather than relying on unverified text edits. In multi-Trojan designs, candidates are introduced and checked iteratively so that each embedded Trojan remains functional without breaking baseline functionality. We have clarified how trigger and payload checks are performed and summarized in the revision.
A4
We appreciate this practical point regarding compute cost and component attribution. FindHT adds structured evidence gathering, hypothesis checking, adversarial challenge, format projection, and fault recovery (Appendix B), which naturally increases token and time usage compared with unguided single-pass runs. In the revision, we clarify the role of format projection and fallback recovery, which prevent valid reasoning from being lost due to strict JSON formatting errors, and add a detailed discussion of cost trade-offs and component contributions.
R1 for weakness 1
We have standardized all metric definitions, denominators, and
Fail (%)labels across the main text, Table 1, and the appendix tables in the revision, and added the exact five-run andPass@5aggregation equations in Appendix E.2.R2 for weakness 2
We have expanded the explanation of sample verification and Trojan validation in Section 3.2 and Appendix C, and clarified how FindHT separates structured reasoning and evidence checks from formatting and fault recovery in Section 5.3 and Appendix B.