- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MOLE as both a benchmark for insider-threat detection and an environment for developing better monitors. The authors use an automated research agent to search for monitoring strategies. These strategies combine model judgments with statistical features, including unusual actions and action sequences. The results show improvements over a uniform GPT-4.1 monitor across multiple corpora. Also, using a stronger model also improves detection relative to applying that model uniformly at a similar modeled cost.
Critical strengths
The experimentation is fairly broad. I liked that normal and harmful activity in each corpus come from the same generator. the time-based splits and the tests on unseen accounts also make sense to me. The authors also check whether harmful objectives were actually completed rather than treating every assigned attack as successful.
Critical weaknesses
a. The automated search only uses a small validation set. the research agent tries 185 ideas, but it checks them on just three validation days and it can see the overall validation score again and again. this makes it easy to end up with strategies that only work well on those few incidents. b. The human check also covers only a few different examples. there are 540 outcome ratings, but they come from just 32 unique account-days. the raters agree a lot, which is good, but the sample has no refusal cases. so this mainly shows the labels are right for these samples. it doesn’t show they are reliable for every generator, threat and outcome type. c. it’s also not shown that this holds up against attackers who adapt. the stealthy prompts tell agents to blend into normal work, but the attackers never get feedback from a specific monitor. so I still don’t know how well detection works when attackers actively learn to get past the defender.
Questions for authors
- How well do the monitors transfer to a different organization configuration or a held-out threat family?
- Do the main rankings and threat-specific findings remain stable across independent simulation runs?
- Could you report detection before harm including time to the first alert and campaign-level recall?
- How does monitor performance change when attackers can adapt using feedback from the defender?
Summary of revisions and reviewer discussion
We thank the reviewers and Area Chair for their time and consideration. As the discussion period concludes, we wanted to briefly summarize the revisions and subsequent reviewer feedback.
In response to the reviews, we substantially expanded the evaluation with stronger classical baselines (ADSAGE, LAN, and supervised TF-IDF), activity-matched comparisons, held-out-threat evaluation, independent-run stability analysis, additional human validation, campaign-level early detection, and a live adaptive attacker–defender pilot. We also strengthened the paper's positioning relative to prior benchmarks and clarified several methodological details.
We appreciate the positive feedback following these additions. Reviewer UADf increased their rating from 2 to 3, noting that the remaining concerns do not substantially detract from the contribution. Reviewer AgSN confirmed satisfaction with the rebuttal while maintaining their rating of 3. Reviewer 3UJ5 also rated the paper 3.
We hope these revisions and the discussion help inform the Area Chair's final assessment, and we thank everyone again for their constructive feedback.