acceptodds
Withdrawn from ICLR 2027

TrustMe: A Benchmark for LLM-Aided Hardware Design Code Auditing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The advancement of Large Language Models (LLMs) has accelerated the adoption of vibe coding in chip design, catalyzing the emergence of LLM-Aided Design (LAD) as an agile development paradigm now widely deployed in practical chip development workflows. This work focuses on a critical task in the chip supply chain: design code security auditing, with a primary emphasis on hardware Trojan detection. Identifying security threats at the early stages of the chip lifecycle is of paramount importance. Existing LAD research has predominantly centered on code generation, verification script synthesis, and debug completion, with limited attention paid to code security alignment and hardware design security awareness. This gap may lead LLMs to generate or overlook hazardous design code, thereby contaminating downstream segments of the supply chain. To bridge this gap, we present TrustMe, a comprehensive benchmark for hardware code Trojan security auditing. TrustMe comprises 816 tasks sampled from real-world projects, with over 1560 complete and stealthy Trojans implanted. It spans evaluation scenarios from module-level to System-on-Chip (SoC) level, assessing full-stack hardware code security monitoring capabilities, including knowledge-based question answering, vulnerability localization, and attack type analysis. The benchmark supports simulation testing across diverse vibe coding environments, including both agentic and pure-LLM setups. We evaluated 10 mainstream LLMs and 6 agentic tools on TrustMe. Results show that even the best-performing combination achieves a hardware Trojan identification success rate below 50%, underscoring the urgent need to prioritize security considerations in LAD. Furthermore, based on our evaluation findings, we propose a Skill-Powered Trojan detection and identification framework. Operating in a self-iterative improvement mode, this framework achieves an average 12.6% performance boost without modifying existing models or agentic tools. We hope this work will encourage LLM-aided chip design from productive into genuinely trustworthy.

Primary area:
datasets and benchmarks
Keywords:
Hardware Trojan, LLM-Aided Design, Evaluation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v7P2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TrustMe, an LLM-driven benchmark designed to evaluate code auditing and hardware Trojan detection in realistic hardware design contexts. The benchmark comprises 816 evaluation tasks with approximately 200,000 lines of code. It is structured into four subsets: Hardware Security Understanding, Atomic Capability Verification, Module-Level Auditing, and System-Level Auditing. The paper further employs a self-evolving agent-human collaborative workflow to generate stealthy, triggerable Trojans, validated by commercial tools and simulation. The paper evaluates 6 LLMs across 5 agent frameworks, observing that the best configuration only achieves a 37.5% pass rate. Thus, the paper proposes FindHT, a lightweight enhancement framework that improves the average pass rate by up to 19.83%.

Critical strengths
  1. As LLMs become pervasive in RTL generation, their ability to audit and detect hardware Trojans is a critical security bottleneck. TrustMe fills a significant gap in existing benchmarks.
  2. This work builds an self-evolving agent-human collaborative pipeline and contains 1560 functional Trojan programs, which effectively alleviates the issue of data scarcity in this research direction.
Critical weaknesses
  1. Some data in the benchmark comes from TrustHub and textbooks. The paper mentioned that these data may be included in model pre-training data and affects evaluation results. Therefore, this benchmark cannot avoild the issue of data contaminating.

  2. The paper lacks budget allocation experiments, component ablation studies, and cross-combination generalization analysis. It is still unclear whether the improvement of 12.6% comes from structured reasoning or from more tokens. It can be observed that the proposed method has achieved a certain performance improvement by incurring a higher cost. Whether this performance improvement is worthwhile still requires further consideration.

  3. There seems no comparison with traditional Trojan detection tools, static analysis, or commercial EDA security tools. Without the comparison results, it is difficult to determine the significance of the proposed method.

Questions for authors
  1. Is there any comparison of FindHT with the baselines under the same token budget, the same time budget, and the same number of iterations?
  2. Why is the full-score rate of the M subset always 0? Does this result indicate that the scoring mechanism might be too strict, or that the task definition is unclear?
  3. Does the paper consider trying to fix this issue by changing the format, etc.?
  4. Is there any comparison with traditional Trojan detection tools, static analysis, or commercial EDA security tools.
Reviewer_NV4AOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TrustMe, a benchmark for evaluating LLMs and agentic systems on hardware-design security auditing. It organizes 816 reported tasks into four subsets: hardware-security knowledge questions (H), atomic capability verification (A), module-level auditing (M), and system-level auditing (S). The benchmark is intended to measure knowledge, Trojan detection, attribution, localization, CWE classification, characterization, and remediation. The authors also propose FindHT, a structured iterative auditing framework based on evidence gathering, hypothesis generation, source binding, adversarial challenge, and output-format enforcement.

The paper evaluates 15 model–agent configurations, reportedly with five runs and Pass@5, and reports a best pass rate of 37.5%. FindHT is reported to improve average performance by 12.6%. The benchmark construction is claimed to include detector screening, VCS trigger validation, human review, lint, functional simulation using the original testbench, and logic synthesis.

The benchmark direction is important and the supplement contains substantial source material. However, the submitted artifact does not substantiate the benchmark described in the paper. Its metadata directly conflict with the claimed executable validation pipeline; its released M and S tasks are substantially narrower and more shortcut-prone than the claimed auditing setting; and the reported 816-task, five-run evaluation cannot be traced to the submitted 795-task version.

I also do not find the hardware-Trojan framing novel by itself. Unlike software, where malicious behavior may sometimes be assessed primarily at runtime, RTL modifications can directly affect functionality, area, timing, and power, making hardware-specific validation essential. Moreover, once the Trojan component is set aside, constructing Verilog benchmarks from textbooks and open-source hardware designs has been standard practice since early LLM-for-RTL benchmarks. The paper therefore needs to explain more clearly what is fundamentally new beyond adding Trojan-themed modifications to an established benchmark-construction pipeline.

Critical strengths
  1. Evaluating whether LLM agents can identify security problems in RTL is timely and practically relevant.

  2. The benchmark attempts to cover knowledge, detection, localization, classification, and system-level analysis. The supplement includes source projects, modified RTL, metadata, prompts, private answers, and deterministic localization scoring rather than only demonstration examples.

Critical weaknesses
  1. The submitted results failed to demonstrate the validity of the claimed “real data,” and their metadata directly contradicted the validation claims made in the paper. The paper states that Trojan horse candidates must undergo detector screening, VCS-triggered verification, manual review, and verification based on simulation evidence. Appendix C further specifies that accepted candidate solutions must pass Verilator code checks, functional simulation using the original, unmodified testbench, and logic synthesis. The paper even states that every reported instance passed an executability check.

    In contrast, all 225 module-expansion examples used in the submitted benchmark are logged as follows: - rtl_validation: “not_run”

    • testbench_run: false All ten SoC workspaces also recorded testbench_run: false. Eight of them recorded syntax validation as not_run, while two Chisel projects recorded expansion validation as not_run. Although some selected targets displayed static_validation: passed, static checks do not substitute for the syntax/expansion checks, functional simulation, trigger and payload verification, and synthesis steps claimed in the paper. These labels also fail to provide the independent verification chains described in the paper. The artifact documentation notes that module localization labels are derived from normalized tag differences between the original RTL and the modified RTL, rather than from instance-based expert security annotations. The CWE labels are inherited from metadata generated by the Trojan horse—and the documentation itself notes that this metadata should be subject to review. In one specific case, the golden location contained common rewrites that replaced 0 and 1 with 1'b0 and 1'b1, illustrating that textual change sets do not automatically correspond to semantically necessary trigger/payload regions. These facts do not prove that every sample is invalid, nor do they prove that the authors never conducted follow-up verification beyond the submitted package. However, they do mean that the submitted evidence fails to support the paper’s central claim—namely, that the actual results of the final benchmark correlate with threats that are executable, triggerable, and retain comprehensive characteristics. If the metadata consists of outdated build-phase records, the authors need to identify subsequent verification records and associate them with the exact final file versions.
  2. The published tasks and scoring mechanisms do not measure the comprehensive autonomous auditing capabilities claimed in the paper, and the reported experiments cannot be traced back to the submitted version. The S-extension is described as evaluating multi-trojan SoC audits across modules and abstraction levels. However, the submitted implementation selects only one target per project, includes a description of that specific issue in the prompt, and instructs the system not to report irrelevant suspicious locations. Of the 249 available location candidates, only 10 (one per project) were scored; the remaining 239 were treated as distractors. While this could serve as a useful guided localization task, it is neither a blind test for unknown threats, nor a comprehensive multi-trojan detection, nor a holistic SoC audit. The M extension also contains structural shortcuts. All 225 tasks are positive examples, and the prompt reveals a target filename. Calling the submitted score_module function reveals that:

    • It always predicts has_threat=true, copies the disclosed filename, and provides no CWE or line information, achieving exactly 40% on every task; - Furthermore, reporting the entire disclosed file as the location yields an average score of 57.7627% without the need to analyze hardware security semantics or access private labels. This behavior stems from the scoring weights: threat presence is weighted at 0.1, file at 0.3, line IoU at 0.4, and CWE at 0.2. Since all M tasks are positive samples and the filenames have been disclosed, a score of 40% can be achieved based solely on task structure. This does not prove that a simple baseline model outperforms the model in the report on the complete benchmark, and the core set does indeed contain negative-example tasks. However, it does indicate that a significant portion of the M score is not evidence of an understanding of the trojan, precise localization, or CWE classification. Finally, the evaluation results in the report cannot be linked to the submitted version:

    • The paper reports 816 tasks, including 246 Class A tasks; - The submitted task catalog contains 795 tasks, including 225 core tasks;

    • The validator included with the work successfully verified an internally consistent version of the 795 tasks;

    • The main text lists Claude-5-Opus as the evaluation model, while Appendices E and F specify GLM-5.2; - The paper states that zero-score items are not included in the average score, yet the submitted report code contains scores with a value of zero;

    • When the actual aggregation function is called offline with the scores 1,0,01,0,0 as inputs, the result returned is 33.333%, rather than the expected 100% after excluding zero scores; - Neither the runner nor the reporter explicitly disclosed the aggregation paths for the five independent, complete runs or the Pass@5 run; - The software package omitted the final INDEX.json, manifest files, configuration snapshots, output results for each question, metrics, and judge logs described in Appendix F, which are required to ensure the traceability of results.

Questions for authors
  1. Can the author point to existing validation records related to the hash values of the last 225 module samples and the 10 selected SoC targets? How do the generic fields rtl_validation: “not_run” and testbench_run: false align with the statement that “every reported instance passed executable validation”?

  2. For the samples that were ultimately scored, which of the following steps were actually completed: lint or syntax validation, applicable Chisel specification, standard functional simulation, trigger activation, payload observation, synthesis, and post-synthesis retention? Please distinguish between these verification steps rather than categorizing them all as static verification.

  3. Prior to submitting the evaluation report, was an independent review conducted of the token-based difference labels and the CWE labels provided by the generator? If so, what existing records document this review and link it to the final document?

  4. Is the purpose of the S extension to measure guided detection of a specific, described issue, or comprehensive detection of all backdoors in the SoC? If it is guided detection, which statements and reported results should be interpreted in light of this narrower scope?

  5. Given that the submitted scorer assigns a score of 40% to “constant positive predictions plus disclosed filenames” and approximately 57.76% to “additionally selecting the entire file,” how should the M-score be interpreted?

  6. What is the exact correspondence between the 816 task benchmarks across 246 core tasks listed in the paper and the submitted catalog of 795 tasks across 225 core tasks? Which version was used to generate the main tables and Figure 4?

  7. Which evaluation model and zero-score convention was used to generate Table 1: Claude-5-Opus or GLM-5.2? Does it include scores with a value of zero? Please clearly specify the specific scoring and reporting entry points.

  8. Can the authors provide the existing output results for each question, five independent run logs, configuration snapshots, and summary files to support the described “Pass@5” protocol and FindHT improvements?

  9. Does “Pass@5” mean that the task is counted as passed if any one of five complete, independent agent runs is successful, or does it refer to another aggregation method? How does this differ from infrastructure retries and execution pass rates?

Reviewer_Qf8zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TrustMe, a benchmark for evaluating how well LLMs and coding agents understand hardware security and audit RTL designs for hardware Trojans. It includes 335 knowledge questions and 481 RTL tasks, covering individual capabilities, module-level auditing, and SoC-level analysis. The authors report 1,560 implanted Trojans. The evaluation compares fifteen model–agent configurations. The paper also introduces FindHT, which adds structured evidence gathering, hypothesis checking, adversarial questioning, and output-format controls to the auditing process. The main contribution is a potentially useful evaluation resource, together with a broad study of current auditing capabilities.

Critical strengths
  1. The benchmark addresses a worthwhile gap.
  2. The paper provides a helpful amount of experimental detail.
Critical weaknesses
  1. The scoring definitions do not consistently match the tables. Appendix E.2 defines the failure rate as the fraction of execution failures. Yet Table 10 reports that all 225 module-level tasks were completed and scored, while their failure rate is 94.2%—exactly the complement of the 5.8% full-score rate. That looks much closer to “did not receive full credit” than “execution failed.” The average-score definition also needs clarification. The paper says zero-score answers are excluded, but the true/false results in Table 10 show an average score equal to the full-score rate, at 89.1%. Under binary scoring, excluding zero-score answers would normally give an average of 100%. Along with the conflicting judge-model descriptions and unclear Pass@5 aggregation, these issues make the headline results difficult to interpret.
  2. The evidence does not fully establish sample validity or explain FindHT’s gains. The main text explicitly reports trigger validation, human review, and simulation-based checking. However, the detailed verification gate in Appendix C mainly describes linting, synthesis, and passing the original testbench. Readers still need evidence that each trigger is reachable and that the intended malicious effect actually occurs. FindHT also includes formatting repair and failure recovery, while the scoring protocol can assign zero to answers that violate the JSON format. Those improvements are useful, but the current experiments do not separate them from better security reasoning or additional computation. Budget-matched comparisons and component ablations would make the result much more convincing.
Questions for authors
  1. What are the exact formulas, denominators, and five-run aggregation rules for Pass@5, Pass rate, Full-score rate, Failure rate, and Average score? Could you walk through Table 10 using the actual counts?
  2. Which judge model was used for the main experiments and for FindHT? How closely do its scores agree with human judgments?
  3. Do all implanted Trojans have dedicated trigger tests and checks for the intended malicious effect? Were Trojans in multi-Trojan designs validated individually?
  4. How much of FindHT’s improvement remains under the same token and time budgets, and how much can be obtained through formatting repair alone?
1 comment

We sincerely thank the reviewer for your time, careful reading, and kind recognition that the benchmark addresses a worthwhile gap and provides helpful experimental detail. Below is our point-to-point rebuttal.

A1

We have clarified the metric definitions and aggregation rules in Section 4, Appendix E.2, and the revised manuscript. Of the 816 items, 14 vision-only items are marked unsupported_vision and excluded from the execution denominators for non-vision agentic runs, leaving 802 runnable items (225 in M). For each configuration, we run five independent trials per item. Pass@5 indicates whether at least one of the five runs succeeds on the task, whereas Pass rate in Table 1 is 1 minus the overall execution failure rate (including agent, LLM, or unrecovered infrastructure failures). Full-score rate is the fraction of scored items that achieve 100% of the rubric points. Average score is the mean rubric-normalized score over scored items with non-zero scores, excluding zero-score and execution-failed items. In Table 10 (deepseek-v4-pro + codex on M, 225 items), all 225 items were run, completed, and scored, with 5.8% (13 items) achieving full score. The 94.2% figure in the Fail (%) column of the per-dimension table reflects items that did not achieve full score (100% − 5.8% = 94.2%). We have clarified this in the revised text and table captions to avoid conflating it with execution failure. For book_true_false (129 scored items), 89.1% reflects the rubric score across scored items as computed by our evaluation script, and we have explicitly documented the exact per-item and five-run aggregation formulas in Appendix E.2.

A2

In our evaluation pipeline, glm-5.2 serves as the judge model for rubric scoring, as specified in Appendix E.2 and Appendix F. We have corrected the conflicting mention in Section 4 in the revision. Deterministic checks handle closed-choice and structured fields, while the judge model evaluates rubric criteria such as free-text descriptions. On a sampled subset audited by human hardware security experts, judge scores align closely with human rubric annotations on structured fields and key security points, and we have added a concise agreement summary in the revision.

A3

Every implanted Trojan in our benchmark is constructed and checked through our verification workflow (Appendix C), including syntax/lint checking, simulation against the design testbench, and synthesis checks. These steps ensure design validity and confirm triggerability and payload behavior, rather than relying on unverified text edits. In multi-Trojan designs, candidates are introduced and checked iteratively so that each embedded Trojan remains functional without breaking baseline functionality. We have clarified how trigger and payload checks are performed and summarized in the revision.

A4

We appreciate this practical point regarding compute cost and component attribution. FindHT adds structured evidence gathering, hypothesis checking, adversarial challenge, format projection, and fault recovery (Appendix B), which naturally increases token and time usage compared with unguided single-pass runs. In the revision, we clarify the role of format projection and fallback recovery, which prevent valid reasoning from being lost due to strict JSON formatting errors, and add a detailed discussion of cost trade-offs and component contributions.

R1 for weakness 1

We have standardized all metric definitions, denominators, and Fail (%) labels across the main text, Table 1, and the appendix tables in the revision, and added the exact five-run and Pass@5 aggregation equations in Appendix E.2.

R2 for weakness 2

We have expanded the explanation of sample verification and Trojan validation in Section 3.2 and Appendix C, and clarified how FindHT separates structured reasoning and evidence checks from formatting and fault recovery in Section 5.3 and Appendix B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.