- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper explore the effects of injecting perturbations in the latent space between a frozen BERT-like encoder and a trainable classifier. Various types of noise is injected at a token-level, sentence-level, and combination in order to explore and isolate various effects. One of the main contributions is that the paper isolates noise magnitude and allocation. The authors show that training on certain noise allocations may results in degradations of other allocations. The findings are primarily evaluated on CASIE, a security even-detection dataset.
Critical strengths
- The authors diligently explore the problem of which types of perturbation allocations during training result in robustness during evaluation. By fixing the amount of perturbation budget, the authors aim to isolate the affects of token-level versus sentence-level perturbations. The authors conduct an extensive set of experiments and perform various ablation and control studies to assess the robustness of their findings and isolate the effects of different perturbation allocations.
- The authors demonstrate an observation that the allocation of training perturbations affects robustness under differing evaluation perturbations. The performance of sentence-level and token-level training perturbations flips depending on the evaluation perturbation channel. This suggest that there is no universal best strategy.
Critical weaknesses
- The authors lack clarity and provide insufficient motivation for their methods. There are a multitude of concepts which are ill-explained including "training arms", "cache", and "test allocations". Additionally, the metrics Trigger-I and Trigger-C are not well-defined. The PHEE dataset is also not cited properly in the main part of the paper, despite being a central point for evaluation. Finally, the modeling choices are poorly defined and not sufficiently motivated. It is unclear why SecureBERT and ModernBERT where chosen as the models for exploration, without giving additional details about their size and training procedure. The classifier which was used for training could be described with more detail.
- There is weak significance and limited generalizability. Is not immediately clear how the methods explored here extend to modern fine-tuning approaches. Additionally, despite the focus on cyber security detection events, the paper could benefit from a stronger explanation for why this particular use-case is chosen. Tying together how the methods explored in this paper would benefit cyber security event detection more concretely would have helped piece together the paper.
Questions for authors
- Can the authors please clarify the decision behind choosing SecureBERT as the main encoder of choice? Additionally by ModernBERT was used for additional experiments?
- Can the authors please clarify what concrete failure modes the perturbations are approximating?
- Can the authors please clarify what role "cache" and "finite precision storage" play in the role of their proposed methodology?
4 comments
Addressing Clarity, Significance, and Generalizability
W1. Clarity and definitions
We will bring the following definitions into the main text:
- Training condition and allocation. We will replace “training arm” with “training condition.” At a fixed training budget, denotes the fraction of the total post-normalization squared-displacement budget assigned to the token path. The test allocation independently specifies the corresponding fraction during evaluation. A value of denotes sentence-only perturbation and denotes token-only perturbation. Both representation inputs remain available in every condition.
- CASIE metrics. Trigger F1 evaluates token-level event identification by merging event types into one positive class. Typed F1 additionally requires the correct event type.
- PHEE and its metrics. PHEE (Sun et al., 2022) is a pharmacovigilance event-extraction dataset. Our PHEE experiments evaluate trigger detection: Trigger-I requires exact trigger boundaries, and Trigger-C additionally requires the correct event type. We will cite and define PHEE when first introducing it.
We will also move the classifier description from Appendix C to §3.1. A fixed orthonormal projection maps 768-dimensional encoder states to 128 dimensions, followed by normalization. The sentence input is the -normalized mean of these token vectors. The primary CASIE classifier forms separate 64-dimensional tanh projections of the two inputs, concatenates their elementwise product with the original inputs, and applies a linear layer, GELU, LayerNorm, and an output layer (five event classes plus background for CASIE).
The allocation–channel interaction also persists with a simpler concatenation head that removes the multiplicative feature: at test budget , the interaction is 12.94 Trigger F1 points (95% CI [8.41, 17.46]) (Appendix B.4, Table 23).
W2. Significance and generalizability
-
Encoder updating. [new] We conducted 20 matched ModernBERT training runs across frozen/trainable encoders, two endpoint training allocations, and five paired seeds under a common protocol. At test budget , the Trigger interaction—the change in the relative advantage of the two training allocations between the endpoint test channels—decreases from 13.50 to 1.03 F1 points with encoder updates. The paired reduction is 12.47 points (95% Student- CI [9.52, 15.43]). This identifies encoder updating as a strong moderator of allocation-dependent robustness.
-
Evidence beyond CASIE. [new] PHEE now covers all five TextEE splits, with eight paired seeds per split. At test budget , both Trigger-I and Trigger-C exhibit positive interactions of approximately one F1 point, with all ten split-specific 95% Student- confidence intervals entirely above zero. The smaller PHEE interaction is accompanied by weaker redistribution of path sensitivity. These results consistently reproduce the conditional robustness finding across data partitions and extend it to pharmacovigilance exact-span detection.
-
Cybersecurity motivation and practical significance. Sparse cybersecurity event detection (such as CASIE) is a particularly fitting testbed because threat intelligence reports combine rapidly evolving technical terminology (challenging local token representations) with complex, noisy sentence structures (challenging global sentence summaries) under limited annotation budgets. More broadly, our findings show that equal perturbation budgets can produce different conditional robustness profiles, with aggregate rankings depending on evaluation-channel weights. This motivates reporting robustness across test allocations and making aggregation weights explicit. We will highlight these findings and their implications in the revised Results and Discussion.
Thank you for adding answers to my questions/weaknesses.
I checked the paper manuscript again and did not find the updates you mentioned (i.e. defining PHEE when it is first introduced). Are you planning on adding these before the deadline?
Thank you for following up and for taking the time to check the manuscript again. We have submitted the revised manuscript and incorporated the changes discussed in our response.
We clarified the perturbation settings and evaluation metrics, explained our modeling choices and the motivation for studying cybersecurity event detection, and defined and cited PHEE in the abstract (p. 1, line 025). We also report the matched comparison of frozen and fine-tuned ModernBERT and the results across all five PHEE splits (p. 8).
We hope these revisions address your concerns and make the study’s motivation and the scope of its findings clearer.
Clarifications on Model Choices, Perturbations, and Caching
Thank you for the careful review and for identifying where our presentation and definitions need clarification. We address Q1–Q3 first, followed by W1–W2. Additional experimental results are marked [new].
Q1. Why SecureBERT, and why ModernBERT?
We will make these selection reasons and model specifications explicit in the experimental setup.
Q2. What concrete failure modes do the perturbations approximate?
The probes examine instability in two inputs used to classify each candidate token:
The token representation itself already incorporates context, and natural-text changes can alter both inputs together. Our experiments independently perturb the classifier inputs under a fixed total post-normalization displacement budget to isolate their respective contributions to robustness. We will clarify this interpretation in the Methodology and consolidate the scope of natural-text validation in the Limitations.
Q3. What role do “cache” and “finite-precision storage” play?
Because the encoder is frozen in our primary controlled study, we compute and cache the representations once, allowing all augmentation conditions to reuse identical features without repeated encoder computations. The vectors are stored in
float16and cast tofloat32when loaded. Noise is added to the loaded values before normalization, and displacement is measured relative to the corresponding normalized clean vectors. The calibration and realized-budget checks account for this numerical implementation.To assess the numerical consistency of cached and freshly reconstructed representations, our ModernBERT reconstruction check rebuilt the features directly from the converted texts using the identified encoder weights, tokenizer, and projection, then evaluated them with the trained classifiers held fixed. The maximum F1 difference was 0.0086 points on clean evaluation and 0.0269 points across 270 noisy evaluations, while the positive stress interaction was preserved. These results support the numerical fidelity of the cached evaluation pipeline. We will explain the cache’s practical role briefly in §3.1 and retain the reconstruction and precision details in Appendices A.4–A.5.