- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents McNdroid, a longitudinal multimodal benchmark for Android malware detection under temporal distribution shift. The dataset contains 858,859 applications spanning 2013–2025, with aligned static, dynamic, and graph-based features. The authors evaluate unimodal detectors and multimodal fusion strategies across different temporal gaps, and further analyze feature distribution shifts, cross-modal disagreement, malware-family evolution, and concept drift adaptation. The results show substantial performance degradation over time, while certain multimodal configurations improve detection performance under long-term drift. The authors release the dataset, evaluation splits, and code to support future research on temporal robustness in malware detection.
Critical strengths
- The paper introduces a substantial Android malware dataset, providing a valuable resource for studying long-term distribution shifts and temporal generalization in malware detection.
- Compared with current datasets, McNdroid provides three dimensions of features covering static, dynamic, and graph-based analysis, enabling systematic comparisons across modalities and supporting research on multimodal learning under temporal distribution shifts.
- The authors evaluate multiple detection models and fusion strategies, alongside analyses of feature drift, cross-modal disagreement, and concept drift adaptation, demonstrating the dataset's potential to support diverse research directions.
Critical weaknesses
- There is confounding between temporal drift and class-distribution shifts. The evaluation does not adequately distinguish temporal feature drift from changes in class distribution. As shown in Table4, the proportion of malware samples varies substantially across years. Such variations in proportion between benign and malicious samples can significantly affect detection performance, particularly precision and F1, even without changes in the underlying class distributions. So, the reported performance degradation may reflect both temporal feature drift and class distribution shifts, making it difficult to isolate the impact of malware evolution. The authors should clarify the rationale behind these varying class distributions and conduct controlled experiments with consistent class ratios across years to assess whether the observed temporal degradation remains significant.
- In Section 3.1, the authors state that 2015 is excluded because AndroZoo contains too few valid entries. I understand it refers to existing work, but earlier studies, such as TESSERACT (USENIX Security 2019), also collected Android applications from AndroZoo and included samples from 2015. The authors should clarify whether this limitation originates from the AndroZoo data source, the specific data collection procedure, or subsequent labeling and feature extraction filters. Providing the number of available samples and the reasons for their exclusion would help justify this gap in the longitudinal benchmark.
- The claimed robustness improvements from multimodal fusion are not consistently supported by the results. As shown in Table7, the best fusion strategy varies across models, and combining all three modalities does not consistently outperform pairwise fusion. Moreover, Figure 3 shows negative fusion gains in 2024 and 2025, suggesting that multimodal fusion does not necessarily improve robustness under long-term temporal shifts. More importantly, Table 2 reports the best fusion configuration for each model-split pair, but it is unclear how these configurations are selected. If the best configuration is chosen based on performance on the test set, the reported improvements may be overly optimistic. The authors should clarify the selection protocol and evaluate whether fixed fusion strategies, selected using only historical validation data, provide consistent improvements under temporal distribution shifts.
- I think the benchmark evaluation lacks representative Android malware detection baselines. The main experiments primarily evaluate generic classifiers, rather than published or the latest Android malware detection approaches. This makes it difficult to assess how existing domain-specific detectors perform on McNdroid and whether the observed temporal degradation and multimodal benefits generalize beyond basic classifiers. The authors should consider including representative existing malware detection methods, particularly those designed for temporal robustness or multimodal analysis, or explain why such comparisons are not feasible under the proposed benchmark.
Questions for authors
- How were the malware-to-benign ratios determined for each year? Table 4 shows substantial variations in class distribution, even among earlier years with sufficient malware samples. While I understand that 2024 and 2025 contain relatively few malware samples, what explains the variations in earlier years? Could the authors conduct additional experiments with a consistent class distribution across years to distinguish temporal feature drift from class-prior shifts?
- Why is 2015 excluded from the dataset? The paper attributes this to insufficient valid entries in AndroZoo, but previous studies using AndroZoo, such as TESSERACT, include samples from 2015. Could the authors clarify the specific reason for this discrepancy?
- How is the year of each APK determined? Is it based on the VirusTotal first submission date, dex date, or another source? Different timestamp definitions may lead to different temporal distributions and potentially affect the evaluation results. Could the authors clarify the timestamp selection and its reliability?
- In Table 3, feature fusion consistently underperforms static features across HCC, CADE, and DAN-MMD in all temporal settings. What explains this degradation when additional modalities are incorporated, and what does it suggest about the effectiveness of multimodal representations for concept drift adaptation?
- Figure 3 shows that multimodal fusion provides negative F1 gains in 2024 and 2025, despite generally positive gains in earlier years. What causes this reversal? Is it related to changes in class distribution, modality-specific feature quality, or increasing cross-modal disagreement? A more detailed analysis would help clarify the conditions under which multimodal fusion improves or harms temporal robustness.
- The dynamic features are extracted from VirusTotal sandbox reports rather than independently executed APKs. How do the authors ensure the quality and reliability of these features? In particular, how is behavioral coverage assessed, given that a valid sandbox report does not necessarily indicate that malicious behaviors were triggered? Are the sandbox environments and execution conditions consistent across years, and how do the authors validate that the extracted dynamic features are comparable over time?
11 comments
Reviewer HCKt #2
Dynamic Features (Q6): We used VirusTotal dynamic behaviour summary API which merges execution traces from multiple in-house Android sandboxes (Droidy, R2DBox, and Zenabox Android, spanning Android 4.4 to 13) alongside third-party engines like Tencent HABO and Dr.Web VxCube [8]. This diversity allows applications with different Android version requirements, as not all APKs can execute across all OS versions. Although these sandbox environments have evolved over time, VirusTotal normalizes their reports into a common behavioral schema, including but not limited to permissions_checked, sms_sent, files_dropped, and dns_lookups. To assess behavioral coverage, we find that 88.49% APKs exhibit activity in at least two behavioral categories, while 52.63% APKs exhibit activity across all four categories. We will acknowledge that sandbox analysis cannot guarantee the execution of all malicious behaviors and clarify this limitation in the revised manuscript.
Please let us know if you have any further concerns, we will do our best to address those as well.
Please see the references provided in our response to Reviewer MWhC.
Follow-up on Our Responses and Revisions
We have done our best to address your concerns in our responses and revisions. Before the discussion period ends, please let us know if any concerns remain or if further clarification would be helpful.
Reviewer HCKt #1
We thank the review for the feedback.
Class ratios (Q1)
The malware to benign ratios are not manually imposed. It reflects the yearly composition of AndroZoo after applying our labeling criteria, deduplication, and multimodal consistency filtering. The variations in earlier years arise from differences in the number of eligible benign and malware samples retained after these steps. Following prior malware benchmarks [1,4,5], our goal was to maintain relatively balanced class coverage to preserve malware family diversity and temporal coverage. As per reviewer's suggestion, we did class balanced (50:50 ratio) experiments and compared it with original results. We observe similar temporal degradation even with a balanced class ratio, so it clearly shows that feature drift contributes to the performance decline. We will include this analysis in the revised version as well.
Exclusion of 2015 and Timestamp (Q2-3)
The reviewer is correct that TESSERACT [2] includes applications assigned to 2015 because they used “dex_date” in their work. However, the reliability issues with this timestamp are documented by the same research group in Beyond the TESSERACT [7] and in AndroZoo's official documentation [6], which reports that many Google Play applications contain default 1980 timestamps. Following prior work [1], we instead use AndroZoo's added field, which records when an APK entered the repository, to determine each application's year. We exclude 2015 because no samples were available for that year based on the AndroZoo added timestamp. Prior work encountered the same issue and similarly excluded 2015 from its evaluation [1]. We therefore exclude 2015 to maintain a consistent and reproducible temporal evaluation protocol.
Representative methods (Q4) Following the established longitudinal Android malware benchmark [1], we use the same set of models to ensure comparability. We further evaluate three representative concept drift adaptation methods (HCC, CADE, and DAN-MMD). For multimodal evaluation, we adopt established off the shelf fusion approaches, as standardized methods for multimodal adaptation under long-term Android malware drift remain limited. As a dataset and benchmark contribution, McNdroid provides a large-scale dataset and reproducible baselines for evaluating temporal robustness and multimodal learning. Our work establishes a foundation for future research on specialized multimodal detection and adaptation methods.
Negative gains in 2024-2025 (Q5) The negative fusion gains in 2024 and 2025 coincide with two important changes. First, these years contain substantially fewer malware samples, and similar performance degradation is also reported in prior work [1]. Second, our cross-modal analysis (Figures 7 and 8) shows declining agreement among modality-specific classifiers, with dynamic predictions disagreeing most with the other modalities in 2024-2025. Under these conditions, conventional fusion can combine conflicting signals and amplify errors. Figure 4 further shows that fusion improves performance for only 33.3% and 50.0% of models in 2024 and 2025. These findings show the importance of model and fusion strategy selection. These results demonstrate that conventional fusion is not uniformly effective under temporal drift and motivate further research on adaptive fusion methods that account for modality-specific reliability. McNdroid provides the benchmark and reproducible baselines to support such research.