acceptodds
Under review as a conference paper at ICLR 2027

Is a Screenshot Worth a Thousand Tokens? Learning Expert Workflows from SOC Investigations

Abstract

Expert computer workflows contain organizational knowledge that is difficult to preserve and reuse. We investigate whether reconstructing these workflows requires visual inputs or whether compact textual representations can support comparable understanding at lower inference cost. We introduce SOCBench, a benchmark of 86 recorded security operations center (SOC) investigations with paired visual and DOM-based textual representations. Our evaluation framework separately assesses procedural fidelity and summary quality, with reliability examined through cross-judge comparisons and domain-expert assessment. We compare two model families under visual and textual inputs, teacher-supervised fine-tuning, and alternative training-task mixtures. Fine-tuning improves both model families, Qwen-3.5 and Gemma-4, on held-out investigations. For the selected configurations, DOM provides similar observed summary quality while Frames achieves higher procedural fidelity. DOM also reduces mean report-generation latency by 59% on the validation cohort and consumes fewer input tokens. Compact text can therefore support efficient workflow documentation, while visual input retains value for preserving procedural detail.

Primary area:
datasets and benchmarks
Keywords:
Instruction Finetuning, Cyber Security, SLM, Benchmark, Visual Representation, UCI, Expert-level Evaluation, Long-form Evaluation, Web Navigation, Multimodal LLM, UI Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vrVBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SOCBench containing 86 recorded security operations center investigations. It studies if screenshots or structured DOM-text representations better support workflow reconstruction. The authors propose a training pipeline, and results show that DOM representations are comparable with 59% lower report generation latency, but screenshots preserve more procedural information.

Critical strengths
  1. The authors do a good job of motivating how to construct expert workflows from recorded computer interactions.
  2. Provide a benchmark of 86 samples with paired visual and DOM representations from experts.
  3. Show 59% reduction in latency, 45.6% fewer tokens using DOM, but retaining competitive summary quality.
Critical weaknesses
  1. Limited size for generalization: 86 investigations from 5 analysts, only 16 held out.
  2. Visual and DOM methods use different training mixtures, no controlled comparison. Difficult to isolate benefits of representation choice alone.
  3. Eval relies heavily on LLM based judgements, and improvements exclude representation construction costs. The full advantage is very unclear.
Questions for authors
  1. How much of the observed performance difference is attributable specifically to DOM versus screenshots?
  2. What evidence supports the applicability of the findings to other expert workflows?
  3. Are modality differences statistically reliable on the held-out set?
2 comments

We thank the reviewer for the thoughtful comments and for recognizing the value of paired expert recordings and the observed inference-efficiency benefits of DOM representations. We address three aspects below: dataset scale and generalizability; the contribution of input representation; and evaluation reliability, statistical uncertainty, and efficiency.

1. Dataset Scale and Generalizability

SOCBench contains 86 complete, expert-performed investigations that capture long, end-to-end workflows. Collecting these recordings requires domain experts, access to the SOC environment, and substantial analyst time.

Each investigation also yields many task-level training examples: the recorded workflow is divided into sequential batches that form separate reconstruction examples, complemented by investigation-level filtering and summarization examples.

The data were partitioned by primary incident ID before constructing training examples, ensuring that repeated recordings of the same primary incident remain in one split. The resulting partitions contain 53 training, 17 validation, and 16 held-out investigation recordings.

A key motivation for SOCBench is that the methodology extends naturally beyond the SOC setting. The framework starts from recorded computer workflows and derives a domain-specific taxonomy that provides a common language for actions, targets, and transitions. Once such a taxonomy is available, the same pipeline can be used to represent and reconstruct other computer-mediated expert workflows.

We chose SOC investigations as a particularly demanding testbed: workflows involve multiple users, incidents, entities, and information-rich screens, with frequent navigation across views and many interdependent investigative actions. Successfully reconstructing workflows in this setting provides encouraging evidence for the broader applicability of the methodology. Because the framework derives its action taxonomy from the workflows of the target domain, rather than relying on a fixed set of SOC-specific actions, the same methodology can in principle be adapted to other computer-based workflows. We see this as a promising foundation for broader expert computer workflows and plan to evaluate the methodology empirically across additional domains, including productivity and creative applications.

2. Isolating DOM Versus Screenshots

To examine the contribution of input representation more directly, we conducted an additional comparison between DOM and Frames without generic auxiliary tasks. Both configurations use 381 Defender investigation examples. The examples are a subset of the original 530-example pool, with examples exceeding the permitted training length under the DOM representation excluded.

This comparison controls the number of Defender investigation examples and the task mixture, providing a more direct assessment than the main comparison between independently selected configurations. We will document the retained-example correspondence and training settings explicitly.

Rubric Comparison Under Matched Training Conditions

The following table reports section-level Rubric scores under the refined evaluation protocol. Positive differences indicate an advantage for DOM.

Rubric sectionValidation DOMValidation FramesΔ (DOM − Frames)Test DOMTest FramesΔ (DOM − Frames)
Title100.0099.71+0.2993.5595.62−2.07
Incident description89.8081.96+7.8485.4384.58+0.85
Detected evidence83.0580.99+2.0680.9080.12+0.78
Conclusion85.8877.06+8.8268.7568.12+0.62
Narrative86.8176.67+10.1579.2279.90−0.68
Overall Rubric87.9081.38+6.5281.5781.24+0.33

Scores are percentages. Evaluation covers 17 validation and 16 held-out investigations.

DOM leads on validation rubric quality by 6.52 points, with its largest section-level advantages in Narrative and Conclusion. On the held-out set, the overall rubric scores are close, with DOM ahead by 0.33 points. DOM scores slightly higher in Incident Description, Detected Evidence, and Conclusion, while Frames scores higher in Title and Narrative. Procedural-fidelity evaluation favors Frames, with PF-F1 scores of 84.68 versus 80.25, a 4.43-point advantage.

These findings support competitive observed summary quality under closer training conditions, while showing that a procedural-fidelity gap persists. The closeness of the rubric means is descriptive and should not be interpreted as a formal demonstration of equivalence. We will report this comparison separately from the selected-configuration results and clarify the extent of experimental control in the revised paper.

3. Evaluation Reliability, Statistical Uncertainty, and Efficiency

Evaluation Reliability

Our framework combines LLM judgments with structured evaluation procedures. Procedural fidelity uses evidence-grounded action matching, deterministic correspondence constraints, and taxonomy-based scoring. Summary quality uses an anchored rubric with fairness review. Repeating GPT-5.4 scoring five times on fixed model outputs showed low variability, supporting evaluator repeatability.

We additionally assessed alignment with SOC-expert judgments. The expert study covered 11 investigations and three archival model configurations, with the automatic judges reproducing the aggregate human ordering on that subset. This provides supporting evidence for the evaluation framework.

Efficiency and Representation-Construction Costs

We agree that the measurement boundary should be explicit. The reported 59.0% latency reduction and 45.6% reduction in cumulative input tokens concern the measured report-generation pipeline on the validation cohort. These measurements exclude decoding and representation construction.

In our implementation, representation construction is CPU-based and does not require GPU inference. Both representations can be constructed online while the investigation is in progress, allowing this work to overlap with analyst activity and potentially reducing additional post-investigation waiting time. However, CPU-based construction still consumes resources, and overlap does not establish that its cost is negligible.

Statistical Uncertainty

We assessed uncertainty across the 16 held-out investigations using paired analyses, averaging five repeated judgments per investigation before analysis. The selected Frames configuration achieved a 4.58-point PF-F1 advantage over DOM (95% paired-bootstrap CI [0.64, 8.69], exact two-sided sign-flip p = 0.049), while its 1.08-point Overall Rubric advantage was not statistically significant (95% CI [−1.65, 3.82], p = 0.464). These unadjusted results suggest a borderline advantage for Frames in procedural fidelity, but no statistically significant difference in summary quality.

Given the limited cohort of 16 investigations, which includes complex trajectories involving repeated hunting queries, incident transitions, and revisits to earlier evidence, we interpret these differences cautiously (see table below). Furthermore, because the selected configurations differ in training volume and auxiliary supervision, the observed differences reflect their combined configurations rather than input representation alone.

Table 2. Investigation characteristics across validation and held-out test sets.

CharacteristicValidationHeld-out test
Investigations1716
Mean recording span (min)19.599.61
Mean captured frames228.41124.06
Mean recorded page visits13.8813.31
Page visits per 100 frames (pooled)6.0810.73
Mean user profiles inspected0.350.56
Maximum user profiles inspected14

Values are means per investigation unless otherwise indicated. Page visits count distinct recorded Page IDs, not unique URLs; navigation density uses pooled page and frame counts. Recording span includes idle time.

Taken together, these findings support DOM as a competitive quality-efficiency alternative: the observed quality differences between the selected models remain statistically uncertain on the held-out cohort, while DOM demonstrates substantial reductions in inference requirements. We will report the paired uncertainty analysis and clarify the efficiency measurement boundaries in the revised manuscript.

Reviewer_FXxtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to turn recorded expert computer work into reports. SOCBench contains 86 investigations by five SOC analysts in a Defender demo environment. Each recording has screenshots and a DOM-based text view. The authors train Qwen and Gemma models using teacher outputs and several training tasks. They score both the report quality and how well the report follows the analyst's actions. With the fine-tuned Qwen setup, DOM needs fewer tokens and less report-generation time, while Frames keeps more procedural detail.

Critical strengths

I vote for the paired recordings and the separate scores for reports and actions. A fluent report can still miss important steps. I think this distinction is useful. The long expert recordings also add something beyond short web tasks.

Critical weaknesses
  • I do not find the main comparison fair enough to isolate the input format. The screenshot and DOM models use different task mixtures, with 4,220 and 487 training examples. Their transcription sets also differ: 530 vs 381 examples. The authors mention matched tests, but Appendix A gives too few scores for those tests and the task-mixture ablations.
  • I find the example report worrying. In Appendix B.1, it rules out an endpoint foothold after process and network queries find no records. But InitiatingProcessRemoteSessionIP and ProcessRemoteSessionIP concern RDP sessions. Empty results do not rule out endpoint compromise as a whole. The timeline also puts 06:31:37 after 06:32:29. I need to know whether this is a reference or an output, and which timestamp rules apply. The current example raises doubts about the checks on report quality.
  • I am unsure how independent the test cases are. The split keeps each recording in one set. But different analysts may investigate the same incident or scenario. The paper does not make clear whether these cases cross the splits. If they do, the 16 test recordings may include familiar cases with different actions.
Questions for authors
  • Can you provide the quality scores from the matched training comparison and the task-mixture ablations mentioned in the text? I would like to see which effect remains after those controls.
  • Is Appendix B.1 a frozen reference or a model output? What supports the endpoint conclusion beyond the empty queries, and what explains the reversed timestamps?
  • How many unique incidents and scenario scripts are in each split? Can the same incident or script appear in recordings from different splits?
2 comments

We thank the reviewer for the constructive feedback and for recognizing the value of paired expert recordings and the separate evaluation of summary quality and procedural fidelity.

1. Fairness of the DOM–Frames Comparison

Our central objective is to investigate whether a substantially more compact textual representation can support competitive summarization quality relative to a strong vision-based alternative. We therefore selected a strong Frames configuration through validation-based model selection, completed before held-out evaluation. The main comparison assesses the practical quality–efficiency trade-off between these selected systems. We agree that examining the contribution of input representation separately requires additional controls.

The selected Frames configuration uses 530 Defender investigation examples together with auxiliary supervision from T1–T3. The DOM configuration uses only 381 Defender investigation examples and no auxiliary supervision. These 381 examples are a subset of the original 530-example pool: examples whose DOM representations exceeded the permitted training length were excluded. These counts refer to training examples derived from the investigation recordings.

We expanded the ablation analysis to examine Frames configurations with 530 or 381 Defender investigation examples, both with and without auxiliary supervision. Alongside these experiments, we refined the evaluation protocol to make the judging criteria more consistently specified and applied. All configurations below were evaluated under the same refined protocol. The configurations used in the main comparison remain those selected through the original validation-based procedure.

The Qwen-3.5-4B results are:

Table 1. Qwen-3.5-4B DOM–Frames comparison and training ablations. All configurations include the primary tasks T5–T7. Scores are percentages, evaluated on 17 validation and 16 held-out test investigations. The highest score in each metric is highlighted in bold.

ConfigurationDefender investigation examplesAuxiliary tasksValidation rubricValidation PF-F1Test rubricTest PF-F1
Frames, selected530T1–T386.82 ± 0.5590.81 ± 0.4082.65 ± 0.7784.84 ± 0.86
Frames530None83.66 ± 1.0089.91 ± 0.2382.13 ± 0.7885.76 ± 0.35
Frames381T1–T384.61 ± 0.3090.76 ± 0.4383.67 ± 0.4486.39 ± 0.67
Frames, reduced381None81.38 ± 0.6389.44 ± 0.6281.24 ± 0.6784.68 ± 0.79
DOM, selected381None87.90 ± 0.5685.75 ± 0.6781.57 ± 0.3880.25 ± 0.63

Selected-model comparison. DOM achieves a higher validation rubric score than the selected Frames baseline, 87.90 versus 86.82. On the held-out set, Frames leads on the rubric by 1.08 points and on PF-F1 by 4.59 points.

We acknowledge that the refined evaluation changes the held-out rubric ordering: the submitted protocol yielded 84.24 for Frames and 84.97 for DOM, whereas the refined protocol yields 82.65 and 81.57. We will document the protocol changes and report their effect explicitly. Our interpretation therefore emphasizes the observed quality–efficiency trade-off.

Comparison under closer training conditions. To reduce differences in training-pool size and task mixture, we compare DOM with Frames trained on 381 Defender investigation examples without auxiliary tasks. DOM leads on validation rubric quality, 87.90 versus 81.38. On the held-out set, rubric scores are close: 81.57 for DOM and 81.24 for Frames, a 0.33-point difference favoring DOM. Frames retains higher procedural fidelity, 84.68 versus 80.25.

This comparison supports similar observed summarization quality under more comparable training conditions, alongside a persistent procedural-fidelity gap. We will specify the exact correspondence between the retained training examples and document any remaining differences in supervision or optimization, so that the extent of experimental control is explicit.

Contribution of auxiliary supervision. The benefit of auxiliary tasks depends on the training configuration and evaluation metric. With 381 Defender investigation examples, auxiliary supervision improves held-out rubric quality by 2.43 points and PF-F1 by 1.71 points. With 530 examples, the rubric improvement is smaller, at 0.52 points, while PF-F1 decreases by 0.92 points. Both settings show improved validation rubric scores.

2. Appendix B.1: Interpretation of the Evidence and Timestamp Ordering

We thank the reviewer for highlighting these points. We agree that empty results for InitiatingProcessRemoteSessionIP and ProcessRemoteSessionIP, on their own, cannot establish the absence of endpoint compromise. We would like to clarify how this example should be interpreted in the context of the recorded investigation and the scope of our benchmark.

The report in Appendix B.1 reconstructs the investigation performed by the analyst; it does not conduct a new, autonomous investigation. SOCBench is designed to recover the observed expert workflow, including the analyst's actions, observations, and documented conclusions, rather than independently determine an optimal investigation strategy or substitute its own judgment for that of the analyst.

In this case, the report states that the absence of DeviceProcessEvents/DeviceNetworkEvents "indicates no endpoint foothold, constraining scope to cloud access." The term "indicates" reflects the analyst's interpretation of the available evidence rather than a definitive claim that endpoint compromise was ruled out. Importantly, this interpretation was informed by the broader investigation, not solely by the two RDP-related fields. The absence of corroborating evidence in the examined endpoint telemetry was considered alongside positive identity-logon and cloud-application activity associated with the suspicious IPs. Together, these observations informed the analyst's decision to narrow the investigative scope toward cloud-identity activity. We agree that the negative RDP-related results alone cannot conclusively rule out endpoint compromise; the example captures the analyst's evidence-driven narrowing of scope rather than making such a general claim.

More broadly, this example highlights an important distinction that we will clarify in the revised manuscript: SOCBench evaluates fidelity to an observed expert workflow, not the independent correctness or optimality of the underlying investigation. Agreement with the reference therefore measures how faithfully the recorded investigation is reconstructed and should not be interpreted as certifying that every analyst judgment was uniquely correct or that no alternative investigative path could have been taken.

Correction of Timestamp Ordering

Regarding timestamp ordering, we confirmed that the apparent reversal of 06:31:37 and 06:32:29 was introduced during preparation of the appendix when the report was copied into the manuscript. This is a presentation error and does not reflect the chronological ordering in the underlying source. We will correct this error in the revised manuscript and provide the corresponding source files so that the original sequence can be verified directly.

3. Independence of Train and Test Investigations

We thank the reviewer for raising this concern. We addressed this during dataset construction by splitting the data at the incident level, using incident IDs rather than investigation IDs.

This ensures that if the same incident was investigated multiple times, including by different analysts, all recordings associated with that incident were assigned to the same split. Consequently, no primary incident appears in more than one of the training, validation, or held-out test sets.

Specifically, the dataset contains 85 unique primary incidents, with 52 assigned to training, 17 to validation, and 16 to the held-out test set. These correspond to 53, 17, and 16 investigation recordings, respectively, as one training incident was investigated twice.

Beyond preventing incident-level information leakage, this design enables us to evaluate the model's ability to generate summaries for previously unseen incidents. The validation and held-out test sets encompass a variety of cybersecurity incident families, providing diverse investigation contexts in which to assess summarization performance.

We will clarify this splitting methodology in Section 3.1 of the revised manuscript, as the current wording does not sufficiently emphasize the incident-level separation.

Reviewer_TUWQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SOCBench, which records experts’ security investigations and provides both screenshot and DOM representations to compare their performance and processing costs. The authors fine-tune Qwen and Gemma using these data, and both model families improve on the test set. For Qwen, the two inputs achieve similar report quality, while DOM enables faster report generation and screenshots preserve more procedural detail.

Critical strengths
  • The research question has practical value. The dataset captures experts’ complete investigation workflows and provides both screenshots and DOM representations for each investigation, allowing comparison of report quality and processing cost across the two inputs.
  • The method shows clear improvements across both model families on the held-out test set. The authors also evaluate summary quality and procedural fidelity separately, making the evaluation more targeted.
Critical weaknesses
  • The ablation results for the support tasks are incomplete. The selected Frames model uses T1–T3 together with T5–T7, but results for training on T5–T7 alone are not reported, making the additional contribution of the support tasks unclear.
  • Methodological novelty is limited. The main contributions are dataset construction, training-pipeline integration, and evaluation design, with substantial engineering value.
Questions for authors
  • Could the authors report a comparison between the Frames model trained only on the primary tasks (T5–T7) and the model trained with the additional support tasks (T1–T3)? This would help me better understand how much the support tasks contribute to the final performance.
2 comments

We thank the reviewer for the thoughtful feedback and for recognizing the practical relevance of our research question, the value of SOCBench, and the importance of separately evaluating summary quality and procedural fidelity.

To clarify the scope of our study, our objective is not to identify the optimal training strategy for vision models, but rather to investigate whether a substantially more compact text-to-text representation can remain competitive with a strong vision-based alternative. Accordingly, we sought to compare DOM against a strong Frames configuration supported by our development experiments. These experiments showed that incorporating auxiliary tasks T1–T3 improved Frames performance relative to training exclusively on the primary reconstruction tasks T5–T7. We therefore selected the stronger, auxiliary-trained configuration for the main comparison, based on validation performance prior to held-out evaluation.

1. Additional Evaluation and Task-Mixture Ablation

To provide a more comprehensive assessment, we additionally re-evaluated the compared configurations using a refined evaluation protocol with stricter judging criteria. The original evaluation protocol remains valid; this refinement provides an additional assessment under more stringent scoring conditions. All results reported below were obtained using the same refined protocol.

Table 1. Frames Qwen-3.5-4B task-mixture ablation. Validation (n = 17) and held-out test (n = 16) results (%), reported as mean ± sample standard deviation across three evaluation repetitions under the refined judging protocol. Higher values indicate better performance.

Frames training mixtureValidation RubricValidation PF-F1Held-out test RubricHeld-out test PF-F1
Primary tasks only (T5–T7), reduced data81.38 ± 0.6389.44 ± 0.6281.24 ± 0.6784.68 ± 0.79
Selected configuration (T1–T3 + T5–T7)86.82 ± 0.5590.81 ± 0.4082.65 ± 0.7784.84 ± 0.86

The results demonstrate that the selected Frames configuration incorporating auxiliary supervision achieves higher Rubric scores on both validation (+5.44 points) and the held-out test set (+1.41 points). Procedural fidelity also improves, although the held-out PF-F1 difference is smaller (+0.16 points). These findings support our selection of the auxiliary-trained Frames configuration as a strong visual baseline. Importantly, because the configurations differ in both auxiliary supervision and the total number of training examples, the observed gains reflect the combined effects of these differences, rather than isolating the contribution of auxiliary supervision alone.

2. Practical Significance: An Illustrative Investigation

Although the aggregate improvement appears modest, individual investigations illustrate its practical relevance. For example, in held-out incident 103953, the source identifies u2227 as the affected user and u22435 as the assigned analyst, while explicitly showing "Mailboxes (0)." Nevertheless, the primary-task-only Frames model incorrectly describes u22435 as an impacted mailbox, introducing both an incorrect entity role and an unsupported affected asset. The auxiliary-trained Frames model avoids this error. Across three GPT-5.4 judgments, the factual-precision criterion D-2 increases from a mean of 3.33/8 to 7/8. GPT-6 judgments show the same direction of improvement. This example illustrates how a modest cohort-level difference can correspond to a consequential reconstruction error in an individual investigation. It does not establish the frequency of such improvements or suggest that auxiliary training uniformly benefits every investigation or evaluation metric.

More broadly, this case illustrates the importance of preserving entity identities and roles when reconstructing expert workflows. Auxiliary supervision may be particularly beneficial for Frames because such information must first be recovered from visual observations. Interestingly, DOM avoids these same errors without auxiliary training, plausibly because much of the relevant information is already available in structured textual form. We do not interpret this individual case as evidence that DOM is error-free or as establishing this mechanism causally. Nevertheless, it is consistent with our broader finding: The DOM model, fine-tuned using only the primary reconstruction tasks, remains competitive with the strengthened Frames baseline in summary quality, while achieving 59.0% lower report-generation latency and requiring 45.6% fewer cumulative input tokens.

3. Methodological Novelty and Scientific Contribution

Regarding methodological novelty, our central contribution lies in making this trade-off measurable for complete expert investigations through paired representations and complementary evaluation methodologies. Our procedural-fidelity framework assesses source-grounded correspondence between investigative actions and their targets, while our summary-quality rubric evaluates the accuracy and quality of the resulting reports. Separating these evaluation dimensions reveals an important finding: similar summary quality can coexist with materially different levels of investigative procedure preservation. Evaluating report quality alone would fail to capture this distinction.

4. Broader Impact and Future Research Directions

Finally, we appreciate the reviewer's recognition of the broader contributions of this work, particularly the challenges of collecting expert workflow data and evaluating the reconstruction of long, complex investigations. We are currently exploring a natural extension of this research: using reconstructed expert workflows as the foundation for agentic execution. Specifically, we are translating the workflow taxonomy into a structured action and tool space, mapping observed analyst operations to executable tools with explicit input/output contracts. This establishes a bridge between reconstructing expert behavior from interaction traces and investigating how autonomous agents can reproduce these workflows through tool use. In this respect, the dataset, workflow taxonomy, and evaluation framework introduced in this work provide a foundation for future research on autonomous investigative agents.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you again for your time and thoughtful feedback on our submission. We are following up on our rebuttal, which includes additional training comparisons, paired statistical analyses, and clarifications regarding the evaluation methodology, dataset partitioning, and efficiency measurements.

We would appreciate your feedback on whether these additions address the concerns raised in the reviews and which points, if any, remain unresolved. We would welcome the opportunity to provide further clarification during the discussion.

Thank you for your consideration.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.