- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SOCBench containing 86 recorded security operations center investigations. It studies if screenshots or structured DOM-text representations better support workflow reconstruction. The authors propose a training pipeline, and results show that DOM representations are comparable with 59% lower report generation latency, but screenshots preserve more procedural information.
Critical strengths
- The authors do a good job of motivating how to construct expert workflows from recorded computer interactions.
- Provide a benchmark of 86 samples with paired visual and DOM representations from experts.
- Show 59% reduction in latency, 45.6% fewer tokens using DOM, but retaining competitive summary quality.
Critical weaknesses
- Limited size for generalization: 86 investigations from 5 analysts, only 16 held out.
- Visual and DOM methods use different training mixtures, no controlled comparison. Difficult to isolate benefits of representation choice alone.
- Eval relies heavily on LLM based judgements, and improvements exclude representation construction costs. The full advantage is very unclear.
Questions for authors
- How much of the observed performance difference is attributable specifically to DOM versus screenshots?
- What evidence supports the applicability of the findings to other expert workflows?
- Are modality differences statistically reliable on the held-out set?
2 comments
3. Evaluation Reliability, Statistical Uncertainty, and Efficiency
Evaluation Reliability
Our framework combines LLM judgments with structured evaluation procedures. Procedural fidelity uses evidence-grounded action matching, deterministic correspondence constraints, and taxonomy-based scoring. Summary quality uses an anchored rubric with fairness review. Repeating GPT-5.4 scoring five times on fixed model outputs showed low variability, supporting evaluator repeatability.
We additionally assessed alignment with SOC-expert judgments. The expert study covered 11 investigations and three archival model configurations, with the automatic judges reproducing the aggregate human ordering on that subset. This provides supporting evidence for the evaluation framework.
Efficiency and Representation-Construction Costs
We agree that the measurement boundary should be explicit. The reported 59.0% latency reduction and 45.6% reduction in cumulative input tokens concern the measured report-generation pipeline on the validation cohort. These measurements exclude decoding and representation construction.
In our implementation, representation construction is CPU-based and does not require GPU inference. Both representations can be constructed online while the investigation is in progress, allowing this work to overlap with analyst activity and potentially reducing additional post-investigation waiting time. However, CPU-based construction still consumes resources, and overlap does not establish that its cost is negligible.
Statistical Uncertainty
We assessed uncertainty across the 16 held-out investigations using paired analyses, averaging five repeated judgments per investigation before analysis. The selected Frames configuration achieved a 4.58-point PF-F1 advantage over DOM (95% paired-bootstrap CI [0.64, 8.69], exact two-sided sign-flip p = 0.049), while its 1.08-point Overall Rubric advantage was not statistically significant (95% CI [−1.65, 3.82], p = 0.464). These unadjusted results suggest a borderline advantage for Frames in procedural fidelity, but no statistically significant difference in summary quality.
Given the limited cohort of 16 investigations, which includes complex trajectories involving repeated hunting queries, incident transitions, and revisits to earlier evidence, we interpret these differences cautiously (see table below). Furthermore, because the selected configurations differ in training volume and auxiliary supervision, the observed differences reflect their combined configurations rather than input representation alone.
Table 2. Investigation characteristics across validation and held-out test sets.
| Characteristic | Validation | Held-out test |
|---|---|---|
| Investigations | 17 | 16 |
| Mean recording span (min) | 19.59 | 9.61 |
| Mean captured frames | 228.41 | 124.06 |
| Mean recorded page visits | 13.88 | 13.31 |
| Page visits per 100 frames (pooled) | 6.08 | 10.73 |
| Mean user profiles inspected | 0.35 | 0.56 |
| Maximum user profiles inspected | 1 | 4 |
Values are means per investigation unless otherwise indicated. Page visits count distinct recorded Page IDs, not unique URLs; navigation density uses pooled page and frame counts. Recording span includes idle time.
Taken together, these findings support DOM as a competitive quality-efficiency alternative: the observed quality differences between the selected models remain statistically uncertain on the held-out cohort, while DOM demonstrates substantial reductions in inference requirements. We will report the paired uncertainty analysis and clarify the efficiency measurement boundaries in the revised manuscript.
We thank the reviewer for the thoughtful comments and for recognizing the value of paired expert recordings and the observed inference-efficiency benefits of DOM representations. We address three aspects below: dataset scale and generalizability; the contribution of input representation; and evaluation reliability, statistical uncertainty, and efficiency.
1. Dataset Scale and Generalizability
SOCBench contains 86 complete, expert-performed investigations that capture long, end-to-end workflows. Collecting these recordings requires domain experts, access to the SOC environment, and substantial analyst time.
Each investigation also yields many task-level training examples: the recorded workflow is divided into sequential batches that form separate reconstruction examples, complemented by investigation-level filtering and summarization examples.
The data were partitioned by primary incident ID before constructing training examples, ensuring that repeated recordings of the same primary incident remain in one split. The resulting partitions contain 53 training, 17 validation, and 16 held-out investigation recordings.
A key motivation for SOCBench is that the methodology extends naturally beyond the SOC setting. The framework starts from recorded computer workflows and derives a domain-specific taxonomy that provides a common language for actions, targets, and transitions. Once such a taxonomy is available, the same pipeline can be used to represent and reconstruct other computer-mediated expert workflows.
We chose SOC investigations as a particularly demanding testbed: workflows involve multiple users, incidents, entities, and information-rich screens, with frequent navigation across views and many interdependent investigative actions. Successfully reconstructing workflows in this setting provides encouraging evidence for the broader applicability of the methodology. Because the framework derives its action taxonomy from the workflows of the target domain, rather than relying on a fixed set of SOC-specific actions, the same methodology can in principle be adapted to other computer-based workflows. We see this as a promising foundation for broader expert computer workflows and plan to evaluate the methodology empirically across additional domains, including productivity and creative applications.
2. Isolating DOM Versus Screenshots
To examine the contribution of input representation more directly, we conducted an additional comparison between DOM and Frames without generic auxiliary tasks. Both configurations use 381 Defender investigation examples. The examples are a subset of the original 530-example pool, with examples exceeding the permitted training length under the DOM representation excluded.
This comparison controls the number of Defender investigation examples and the task mixture, providing a more direct assessment than the main comparison between independently selected configurations. We will document the retained-example correspondence and training settings explicitly.
Rubric Comparison Under Matched Training Conditions
The following table reports section-level Rubric scores under the refined evaluation protocol. Positive differences indicate an advantage for DOM.
Scores are percentages. Evaluation covers 17 validation and 16 held-out investigations.
DOM leads on validation rubric quality by 6.52 points, with its largest section-level advantages in Narrative and Conclusion. On the held-out set, the overall rubric scores are close, with DOM ahead by 0.33 points. DOM scores slightly higher in Incident Description, Detected Evidence, and Conclusion, while Frames scores higher in Title and Narrative. Procedural-fidelity evaluation favors Frames, with PF-F1 scores of 84.68 versus 80.25, a 4.43-point advantage.
These findings support competitive observed summary quality under closer training conditions, while showing that a procedural-fidelity gap persists. The closeness of the rubric means is descriptive and should not be interpreted as a formal demonstration of equivalence. We will report this comparison separately from the selected-configuration results and clarify the extent of experimental control in the revised paper.