- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SpecterRTL to generate hardware Trojans (HTs) in RTL. The approach is a multi-agent workflow: an LLM front-end (DAVE) mines design-specific vulnerabilities from a CWE/CVE knowledge graph and writes seed Trojans, and a genetic search (PHAGE) then evolves those seeds using LLMs as its variation operators (mutate/deepen/recombine), scoring each solution by a weighted sum of a severity score (weight 0.7) and detector evasion (0.3).
Critical strengths
- HT generation is useful for the hardware community, as it might help train better models to detect zero-day HTs.
- The evolutive part of the multi agent workflow has potential to generate unpredicted designs that offer richer HT distributions.
Critical weaknesses
- The central empirical claim (evasion) is a volume artifact, not a real result. The headline / more evasions come entirely from generating far more candidates, not from producing stealthier Trojans. The paper's own rate columns show evasion getting \emph{worse} where the method is applied: the evasion rate falls from to after evolution (Table3a), and is lower than the baseline on the shared designs ( theirs vs for GHOST, Table3b). Section5.3 nonetheless claims a frontier with ``stronger evasion,'' which directly contradicts these numbers. Because detector evasion is one of the two objectives the whole framework is built to optimize, a headline that inverts the sign of the underlying rate is not a presentation nit; it undermines the paper's core quantitative argument.
- The work is LLM-driven HT generation, a space that is already well-populated in the literature, so the ``first end-to-end autonomous'' claim does not hold. The evaluation then measures success as evading HT detectors trained on scarce, unbalanced datasets, a limitation long known in that community, so results establish little w.r.t evading detection. Relatedly, the optimization formulation is decorative and internally inconsistent: the task is posed as a constrained multi-objective problem but it is never solved; it is scalarized to a single weighted sum for selection, while Section5.1 reports Pareto hypervolume as a multi-objective quality measure. Either commit to multi-objective optimization or present the scalarized heuristic plainly.
Questions for authors
- Your headline gains are reported as evasion counts, yet the evasion rate drops after evolution and is below the baseline on the shared designs. How do you reconcile this with the claim in Section~5.3 of a frontier with "stronger evasion,'' and why should counts rather than rates be the figure of merit?
Thank you for asking us to distinguish corpus scale, per-instance evasion, and the role of the optimization formulation.
Counts, rates, and the frontier comparison. Counts measure corpus output, while rates measure the fraction satisfying the detector criterion. On the shared designs, SpecterRTL produces 593 checked instances versus the 14 released by GHOST, including 140 versus 4 instances not flagged by the panel. GHOST's 28.6% rate rests on just 4/14 instances. Changing a single outcome shifts that rate by 7.14 percentage points, exceeding its approximately 5-point difference from SpecterRTL's 23.6%. Such a small baseline does not support a reliable performance ranking from this rate difference, nor does it negate the value of producing a much larger corpus of checked instances, including substantially more detector evasions. Section 5.3 compares non-dominated points against OpenEvolve, whereas Table 3 compares aggregate rates against initial seeds or GHOST. These different comparisons need not move in the same direction. “Stronger evasion” should therefore be tied to the specific comparison it describes, rather than presented as an overall improvement in evasion probability.
Contribution relative to prior generation methods. The contribution is the integration of design-aware seed construction, population-based expansion, and reuse of accumulated examples and feedback. The evidence includes coverage of 137 modules and the ablation yielding 722 checked seeds with the full workflow, compared with 139 without the knowledge graph and 220 with a monolithic agent. This identifies a concrete contribution without relying on a universal “first autonomous” claim. Detector responses are feedback and evaluation observations within this workflow, not a general certification of stealth against arbitrary defenses.
Scalarization and hypervolume. PHAGE uses a weighted scalar fitness to guide heuristic search subject to the design checks. Weighted-sum scalarization expresses a preference over multiple objectives, while hypervolume measures the region dominated by the resulting point set. The latter can be computed regardless of how those points were selected, so the two are mathematically compatible. The appropriate claim is heuristic search under a stated preference, with hypervolume describing the attained set. It is not a claim of recovering the complete Pareto frontier or guaranteeing global optimality.