- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LatentTriad uses a single prefill over a trusted task and a skill package to extract Declaration, Operation, and Boundary states. Three risk branches built from these states yield a package-level verdict before output generation or execution. Across 5,830 skill packages from five sources and seven model backbones, the authors report 99.0%–99.2% pooled F1 and 93.0%–94.6% source-macro F1. Cross-form tests, state interventions, source holdout, and composition experiments examine the method's behavior and limits of generalization.
Critical strengths
- Task-aware, multi-file pre-action skill detection without output generation. The method reads the trusted task and the skill package's readable files, extracts Declaration, Operation, and Boundary states in one prefill, and constructs Boundary, Operation, and Geometry risk representations. It produces a verdict before generating tokens, invoking tools, or executing skill code.
- Broad experimental evaluation. The authors assess package-level detection across five data sources and seven model backbones, with additional cross-form, state-intervention, source-holdout, skill-composition, and branch-ablation studies. These experiments support the main detection result and reveal weaknesses under source shift and synthetic structural transfer.
Critical weaknesses
- Generalization has clear limits. In leave-one-source-out evaluation, mean source-macro F1 falls from 93.9% on the main split to 73.2%. In Appendix G.6, after refitting the full architecture on synthetic data, AUROC is near 50% for both known and held-out composition templates. These results limit extrapolation to new sources and composition structures; the low synthetic performance cannot be attributed solely to template holdout.
- Benign classification is sensitive to operation-preserving rewrites. On the selected 256 matched pairs in Appendix D.1, clean source-macro FPR is already 32.3%–40.8% and rises to as much as 80.4% after region padding and 70.9% after cross-file splitting. These subset results should not be extrapolated to the full data distribution, but false-alarm rates this high would hinder use under the tested conditions.
Questions for authors
- In Tables 16–17, the full fusion rule has slightly lower recall and F1 than at the same FPR, as well as slightly lower AUROC and AP. At which FPR budgets, if any, does Geometry confirmation improve performance? Could you report this comparison with thresholds selected on the development set?
- Could Appendix C.3 report F1, FPR, and recall separately for each of the five held-out sources, to show whether the average decline is driven by particular sources?
- Why is the FPR on unmodified benign packages in Table 22 already much higher than in the main experiment? On the same 256 matched pairs, could you compare the baseline FPR of the Table 2 detector and the Appendix D.1 application probes, then report FPR before and after region padding and cross-file splitting at matched malicious-skill recall?
Reminder: Current Rebuttal Phase
Dear Reviewers,
I hope this message finds you well.
I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.
Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.
If you have any questions or need further assistance, please do not hesitate to reach out.
Best regards,
AC