acceptodds
Under review as a conference paper at ICLR 2027

McNdroid: A Longitudinal Multimodal Benchmark for Robust Drift Detection in Android Malware

Abstract

Machine learning (ML) in real-world systems must contend with concept drift, adversarial actors, and a spectrum of potential features with varying costs and benefits. Malware naturally exhibits all of these complexities, but for the same reason, it is challenging to curate and organize data to study these factors. We present McNdroid, to our knowledge the largest longitudinal multimodal Android malware benchmark for malware detection and drift analysis. McNdroid spans 2013-2025, excluding 2015, and represents each application with three aligned modalities-static features from manifests and smali code, dynamic behavioral features from sandbox execution, and graph-based features from function-call graphs. Using temporally separated splits, we evaluate standard ML and deep-learning detectors across increasing train-test time gaps. Results show clear temporal degradation, while multimodal fusion outperforms the best single modality across long-term temporal gaps. Cross-modal agreement also declines over time, suggesting that drift affects both individual feature spaces and the consistency among modalities. We further analyze modality-specific drift, malware-family evolution, and temporal changes in model explanations. We publicly release McNdroid, benchmark splits, and code to support reproducible research on temporal generalization and robust multimodal learning in security-critical, non-stationary settings. **Data:** https://zenodo.org/records/19969833 **Code:** https://anonymous.4open.science/r/McNdroid-BDEC/

Primary area:
datasets and benchmarks
Keywords:
multimodal malware dataset, cybersecurity, malware detection, malware analysis, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HCKtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents McNdroid, a longitudinal multimodal benchmark for Android malware detection under temporal distribution shift. The dataset contains 858,859 applications spanning 2013–2025, with aligned static, dynamic, and graph-based features. The authors evaluate unimodal detectors and multimodal fusion strategies across different temporal gaps, and further analyze feature distribution shifts, cross-modal disagreement, malware-family evolution, and concept drift adaptation. The results show substantial performance degradation over time, while certain multimodal configurations improve detection performance under long-term drift. The authors release the dataset, evaluation splits, and code to support future research on temporal robustness in malware detection.

Critical strengths
  1. The paper introduces a substantial Android malware dataset, providing a valuable resource for studying long-term distribution shifts and temporal generalization in malware detection.
  2. Compared with current datasets, McNdroid provides three dimensions of features covering static, dynamic, and graph-based analysis, enabling systematic comparisons across modalities and supporting research on multimodal learning under temporal distribution shifts.
  3. The authors evaluate multiple detection models and fusion strategies, alongside analyses of feature drift, cross-modal disagreement, and concept drift adaptation, demonstrating the dataset's potential to support diverse research directions.
Critical weaknesses
  1. There is confounding between temporal drift and class-distribution shifts. The evaluation does not adequately distinguish temporal feature drift from changes in class distribution. As shown in Table4, the proportion of malware samples varies substantially across years. Such variations in proportion between benign and malicious samples can significantly affect detection performance, particularly precision and F1, even without changes in the underlying class distributions. So, the reported performance degradation may reflect both temporal feature drift and class distribution shifts, making it difficult to isolate the impact of malware evolution. The authors should clarify the rationale behind these varying class distributions and conduct controlled experiments with consistent class ratios across years to assess whether the observed temporal degradation remains significant.
  2. In Section 3.1, the authors state that 2015 is excluded because AndroZoo contains too few valid entries. I understand it refers to existing work, but earlier studies, such as TESSERACT (USENIX Security 2019), also collected Android applications from AndroZoo and included samples from 2015. The authors should clarify whether this limitation originates from the AndroZoo data source, the specific data collection procedure, or subsequent labeling and feature extraction filters. Providing the number of available samples and the reasons for their exclusion would help justify this gap in the longitudinal benchmark.
  3. The claimed robustness improvements from multimodal fusion are not consistently supported by the results. As shown in Table7, the best fusion strategy varies across models, and combining all three modalities does not consistently outperform pairwise fusion. Moreover, Figure 3 shows negative fusion gains in 2024 and 2025, suggesting that multimodal fusion does not necessarily improve robustness under long-term temporal shifts. More importantly, Table 2 reports the best fusion configuration for each model-split pair, but it is unclear how these configurations are selected. If the best configuration is chosen based on performance on the test set, the reported improvements may be overly optimistic. The authors should clarify the selection protocol and evaluate whether fixed fusion strategies, selected using only historical validation data, provide consistent improvements under temporal distribution shifts.
  4. I think the benchmark evaluation lacks representative Android malware detection baselines. The main experiments primarily evaluate generic classifiers, rather than published or the latest Android malware detection approaches. This makes it difficult to assess how existing domain-specific detectors perform on McNdroid and whether the observed temporal degradation and multimodal benefits generalize beyond basic classifiers. The authors should consider including representative existing malware detection methods, particularly those designed for temporal robustness or multimodal analysis, or explain why such comparisons are not feasible under the proposed benchmark.
Questions for authors
  1. How were the malware-to-benign ratios determined for each year? Table 4 shows substantial variations in class distribution, even among earlier years with sufficient malware samples. While I understand that 2024 and 2025 contain relatively few malware samples, what explains the variations in earlier years? Could the authors conduct additional experiments with a consistent class distribution across years to distinguish temporal feature drift from class-prior shifts?
  2. Why is 2015 excluded from the dataset? The paper attributes this to insufficient valid entries in AndroZoo, but previous studies using AndroZoo, such as TESSERACT, include samples from 2015. Could the authors clarify the specific reason for this discrepancy?
  3. How is the year of each APK determined? Is it based on the VirusTotal first submission date, dex date, or another source? Different timestamp definitions may lead to different temporal distributions and potentially affect the evaluation results. Could the authors clarify the timestamp selection and its reliability?
  4. In Table 3, feature fusion consistently underperforms static features across HCC, CADE, and DAN-MMD in all temporal settings. What explains this degradation when additional modalities are incorporated, and what does it suggest about the effectiveness of multimodal representations for concept drift adaptation?
  5. Figure 3 shows that multimodal fusion provides negative F1 gains in 2024 and 2025, despite generally positive gains in earlier years. What causes this reversal? Is it related to changes in class distribution, modality-specific feature quality, or increasing cross-modal disagreement? A more detailed analysis would help clarify the conditions under which multimodal fusion improves or harms temporal robustness.
  6. The dynamic features are extracted from VirusTotal sandbox reports rather than independently executed APKs. How do the authors ensure the quality and reliability of these features? In particular, how is behavioral coverage assessed, given that a valid sandbox report does not necessarily indicate that malicious behaviors were triggered? Are the sandbox environments and execution conditions consistent across years, and how do the authors validate that the extracted dynamic features are comparable over time?
11 comments
AuthorsOpenReview ↗

Reviewer HCKt #1

We thank the review for the feedback.

Class ratios (Q1)

The malware to benign ratios are not manually imposed. It reflects the yearly composition of AndroZoo after applying our labeling criteria, deduplication, and multimodal consistency filtering. The variations in earlier years arise from differences in the number of eligible benign and malware samples retained after these steps. Following prior malware benchmarks [1,4,5], our goal was to maintain relatively balanced class coverage to preserve malware family diversity and temporal coverage. As per reviewer's suggestion, we did class balanced (50:50 ratio) experiments and compared it with original results. We observe similar temporal degradation even with a balanced class ratio, so it clearly shows that feature drift contributes to the performance decline. We will include this analysis in the revised version as well.

YearModelUnimodal inputOriginal F1 (%)50:50 F1 (%)Fusion approachOriginal fusion F1 (%)50:50 fusion F1 (%)
2013XGBoostStatic96.9 ± 0.197.5 ± 0.1Feature Fusion97.2 ± 0.097.7 ± 0.0
2014LightGBMGraph86.6 ± 0.088.8 ± 0.3Late Fusion88.0 ± 0.289.9 ± 0.2
2016SVMStatic72.0 ± 0.975.9 ± 0.2Static + Dynamic72.8 ± 0.775.5 ± 1.4
2017DetectBERTDynamic18.8 ± 1.624.5 ± 9.5Static + Dynamic19.6 ± 0.925.6 ± 3.9
2018DetectBERTDynamic18.3 ± 1.021.8 ± 10.4Dynamic + Graph21.5 ± 9.218.8 ± 3.3
2019LightGBMStatic64.0 ± 0.668.8 ± 0.8Feature Fusion73.6 ± 0.678.4 ± 1.1
2020LightGBMStatic59.9 ± 1.366.6 ± 2.6Feature Fusion75.8 ± 0.981.0 ± 0.7
2021LightGBMStatic50.6 ± 0.359.9 ± 1.8Static + Graph67.9 ± 1.774.4 ± 3.7
2022LightGBMStatic54.7 ± 1.162.9 ± 1.9Feature Fusion66.6 ± 1.273.2 ± 1.8
2023LightGBMGraph51.0 ± 0.155.3 ± 0.5Static + Graph54.6 ± 2.762.8 ± 1.6
2024LightGBMGraph53.1 ± 4.264.8 ± 0.6Dynamic + Graph38.9 ± 9.765.4 ± 0.6
2025LightGBMGraph17.7 ± 1.221.5 ± 0.1Dynamic + Graph14.1 ± 1.322.7 ± 0.5

Exclusion of 2015 and Timestamp (Q2-3)

The reviewer is correct that TESSERACT [2] includes applications assigned to 2015 because they used “dex_date” in their work. However, the reliability issues with this timestamp are documented by the same research group in Beyond the TESSERACT [7] and in AndroZoo's official documentation [6], which reports that many Google Play applications contain default 1980 timestamps. Following prior work [1], we instead use AndroZoo's added field, which records when an APK entered the repository, to determine each application's year. We exclude 2015 because no samples were available for that year based on the AndroZoo added timestamp. Prior work encountered the same issue and similarly excluded 2015 from its evaluation [1]. We therefore exclude 2015 to maintain a consistent and reproducible temporal evaluation protocol.

Representative methods (Q4) Following the established longitudinal Android malware benchmark [1], we use the same set of models to ensure comparability. We further evaluate three representative concept drift adaptation methods (HCC, CADE, and DAN-MMD). For multimodal evaluation, we adopt established off the shelf fusion approaches, as standardized methods for multimodal adaptation under long-term Android malware drift remain limited. As a dataset and benchmark contribution, McNdroid provides a large-scale dataset and reproducible baselines for evaluating temporal robustness and multimodal learning. Our work establishes a foundation for future research on specialized multimodal detection and adaptation methods.

Negative gains in 2024-2025 (Q5) The negative fusion gains in 2024 and 2025 coincide with two important changes. First, these years contain substantially fewer malware samples, and similar performance degradation is also reported in prior work [1]. Second, our cross-modal analysis (Figures 7 and 8) shows declining agreement among modality-specific classifiers, with dynamic predictions disagreeing most with the other modalities in 2024-2025. Under these conditions, conventional fusion can combine conflicting signals and amplify errors. Figure 4 further shows that fusion improves performance for only 33.3% and 50.0% of models in 2024 and 2025. These findings show the importance of model and fusion strategy selection. These results demonstrate that conventional fusion is not uniformly effective under temporal drift and motivate further research on adaptive fusion methods that account for modality-specific reliability. McNdroid provides the benchmark and reproducible baselines to support such research.

AuthorsOpenReview ↗

Reviewer HCKt #2

Dynamic Features (Q6): We used VirusTotal dynamic behaviour summary API which merges execution traces from multiple in-house Android sandboxes (Droidy, R2DBox, and Zenabox Android, spanning Android 4.4 to 13) alongside third-party engines like Tencent HABO and Dr.Web VxCube [8]. This diversity allows applications with different Android version requirements, as not all APKs can execute across all OS versions. Although these sandbox environments have evolved over time, VirusTotal normalizes their reports into a common behavioral schema, including but not limited to permissions_checked, sms_sent, files_dropped, and dns_lookups. To assess behavioral coverage, we find that 88.49% APKs exhibit activity in at least two behavioral categories, while 52.63% APKs exhibit activity across all four categories. We will acknowledge that sandbox analysis cannot guarantee the execution of all malicious behaviors and clarify this limitation in the revised manuscript.

Please let us know if you have any further concerns, we will do our best to address those as well.

Please see the references provided in our response to Reviewer MWhC.

AuthorsOpenReview ↗

Follow-up on Our Responses and Revisions

We have done our best to address your concerns in our responses and revisions. Before the discussion period ends, please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_fa35OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces McNdroid, a large-scale benchmark for temporal multimodal Android malware detection. The dataset provided has 858859 samples and 1354 malware families and three modalities (static, dynamic, and graph based). The paper tries to - 1) evaluate distributional shift across time comparing each test year with 2013 training reference 2) understand feature stability across the three modalities 3) evaluate the disagreement between the three modalities.

Critical strengths
  1. Although past works have introduced and looked into the idea of benchmarks for longitudinal android malware analysis, this work introduce a new dimension to the regime which is understanding this problem in multimodal regime.

  2. The program graphs and dynamic traces can be costly to extract at a large scale. Thus this dataset can be useful in filling gap in existing Android malware analysis literature and reduce cost for subsequent works.

Critical weaknesses
  1. The share of malware compared to benign samples drops quite drastically in the years 2024 and 2025. As all the temporal results discuss in the paper deem to amplify in this range as well, it is very difficult to separate the effect of class imbalance from the temporal shifts. It would be helpful to provide malware prevalence-matched comparisons.

  2. The multimodal consistency filtering results in removing around half of the original APK pool dataset. This may disproportionately exclude sandbox-evasive malware families.

  3. The deployment may depend on VirusTotal as the detector needs dynamic representation which would need VirusTotal behavior report for every new APK or an in-house sandbox.

  4. It is not clear to me how the 80-20 split described in section 3.1 is actually used in section 5 evaluations.

  5. Table 2 mixes variation across seeds and years. Per-year results with run-level error bars would be more informative.

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Reviewer fa35

We thank the reviewer for the feedback.

Prevalence versus temporal distribution shift: The lower malware prevalence in 2024-2025 reflects the limited availability of malware samples in AndroZoo for these years. We follow the temporal evaluation protocol established in prior work [1], which similarly evaluates later years under limited malware availability. Importantly, our analysis goes beyond classification performance.

Complete-modality filtering: Our multimodal consistency filtering ensures that all three modalities represent the same applications. It enables controlled comparisons without differences in sample composition. As described in Section 3.1, filtering excludes applications with incomplete static, dynamic, or graph-based extraction, meaning that missing sandbox behavior is only one of several possible reasons for exclusion. Furthermore, sandbox evasion is an inherent challenge in dynamic malware analysis, and no sandbox environment can guarantee complete behavioral coverage. This further motivates McNdroid's multimodal design, even when malware evades sandbox observation, static and graph-based representations can provide complementary evidence of malicious activity. Our benchmark enables researchers to investigate such scenarios through controlled missing-modality experiments, including feature imputation and availability-aware fusion, using identical applications, labels, and temporal splits. We will clarify this motivation in the revised manuscript.

Dependence on sandboxing: McNdroid is designed as a dataset and benchmark for researchers. We do not claim it as a deployable malware detection system. VirusTotal serves as the source of sandbox reports during dataset construction. We release pre-extracted dynamic features which enable researchers to develop and evaluate behavioral and multimodal detection methods without independently collecting costly large-scale sandbox analyses. The benchmark also supports standalone static and graph-based detection, pairwise fusion (Table 7), and missing-modality evaluation (Appendix J). In practical deployment, defenders could prioritize static and graph-based analysis and invoke dynamic analysis when these modalities provide insufficient evidence for a confident decision. Our goal is to provide a standardized, reusable benchmark that enables researchers to investigate such adaptive detection strategies without the burden of large-scale feature extraction.

80/20 split: Following prior work [1,4,5], we do not release a separate validation set. Instead, we partition the original dataset into training (80%) and test (20%) sets within each year and release both partitions.For the temporal evaluation in Section 5, models are trained on the 2013 training partition and evaluated on the test partitions (20%) of subsequent years under IID (2014), NEAR (2016-2017), and FAR (2018-2025) settings.

Per-year results: Following the evaluation protocol adopted in prior work [1], Table 2 reports aggregate performance across IID, NEAR, and FAR settings, with mean and standard deviation computed over five runs and the corresponding evaluation years. Figures 3 and 4 further present year-wise multimodal gains and model-level win rates. In Appendix F, we provide detailed comparisons across fusion strategies. To address reviewers concern, we will additionally include year-wise performance results with error bars in the revised appendix.

Please let us know if you have any further concerns, we will do our best to address those as well.

Please see the references provided in our response to Reviewer MWhC.

AuthorsOpenReview ↗

Follow-up on Our Responses and Revisions

We have done our best to address your concerns in our responses and revisions. Before the discussion period ends, please let us know if any concerns remain or if further clarification would be helpful.

Dear Reviewer,

We have tried our best to address the concerns. If you have any other concerns, we would be happy to address those with our best effort. We really appreciate your support in our paper.

Thank you very much.

Dear Reviewers,  

As the discussion period closes today, we would sincerely appreciate any final feedback you may have on our rebuttal and revisions. If you feel that our responses have adequately addressed your concerns, we would be grateful if you could kindly consider revisiting your assessment and score, if appropriate.  

Thank you very much for your time and effort.

Reviewer_MWhCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces McNdroid, a multimodal benchmark for temporal Android malware detection comprising 858,859 Android samples and 1,354 malware families spanning 12 years (2013–2025, excluding 2015). Each sample is represented using three aligned modalities: static features, dynamic features, and graph-based features. Furthermore, the benchmark is used to perform temporal evaluation under a protocol where the model is trained on 2013 and evaluated in three settings: IID (2014), NEAR (2016-2017) and FAR (2018-2025). Experimental results show temporal degradation across all modalities, with the graph features diverging the most over time. Multimodal fusion helps improve robustness, as cross-modal agreement also degrades over time.

Critical strengths
  • The large multimodal McNdroid benchmark, including three modalities/sets of features (static, dynamic, graph) capturing application artifacts, runtime behavior, and program structure, is a major strength of the paper.

  • The extensive experimental evaluation, with focus on concept drift adaptation under different modalities, cross-modal disagreement, feature stability across modalities, and class distribution, etc., is also seen as a strength.

  • The authors established labels using thresholds on VirusTotal outputs but also released the labels to allow others to work with different thresholds.

  • Data and code are made available anonymously.

Critical weaknesses
  • The class distribution in the benchmark dataset is rather balanced, which does not seem to reflect the actual real distribution of malware to benign, which is known to be very skewed. The authors consider an app benign if the VirusTotal detection/output is 0 and malware if it is greater than 4. Apps with detection scores between 1 and 3 are discarded. Together, these issues make the task easier.

  • The multimodal experiments are performed under several settings (early/late fusion and cross-attention). The main results in the paper correspond to the best fusion configuration for each model–split pair, selected on the test set. There is no separate validation set to select "hyperparameters"/configurations.

  • The results show that static and graph features generally complement each other, but adding dynamic features hurts performance or adds very little in many cases. The authors' claim that the fusion outperforms the individual modalities is not fully supported by the experimental results.

  • The fact that Android app features has changed significantly over the years is well-documented in the literature already. The paper does not have a novel contribution in this aspect.

  • There seem to be some inconsistencies about how features are extracted, with static being extracted from 2013 only (to avoid temporal feature leaking), while dynamic features use hashing and can thus capture newer information. Thus, the claim that static features are most stable and dynamic features disagree most could be partly an artifact of the feature construction.

Questions for authors
  • The authors mentioned an 80/20 split of the data? Where was that used?

  • When were the VT reports retrieved — one snapshot at the dataset construction time, or at AndroZoo scan time? That can influence the malware to benign ratio observed.

8 comments
AuthorsOpenReview ↗

Reviewer MWhC

Thank you for your valuable feedback.

80/20 split: Following prior work [1,4,5], we do not release a separate validation set. Instead, we partition the original dataset into training (80%) and test (20%) sets within each year and release both partitions.For the temporal evaluation in Section 5, models are trained on the 2013 training partition and evaluated on the test partitions (20%) of subsequent years under IID (2014), NEAR (2016-2017), and FAR (2018-2025) settings.

VT retrieval dates: Following prior work [1], the VirusTotal detection counts used for labeling are obtained from AndroZoo metadata. To collect dynamic reports, we submitted a large number of benign APKs to VirusTotal (as they were not available on virustotal) and retrieved their analysis reports during a single collection period. We also performed an additional retrieval of updated VirusTotal detection counts and will release these alongside the original metadata in the final version.

Class balance & VT thresholds: Following established practices in prior malware benchmarks [1,4,5], we prioritize malware family diversity and temporal coverage. Please note that having more malware samples in the benchmark dataset is an advantage for the research community as researchers can partition with 90:10 (benign:malware) very easily. Having more malware samples in the benchmark is not a weakness.

This approach ensures broader representation of malware families which allows researchers to study their behavioral characteristics and evolution over time. Additionally, the relatively balanced dataset allows future researchers to construct different class distributions that reflect deployment-specific scenarios and evaluate models under varying levels of class imbalance. Furthermore, our VirusTotal labeling approach follows established practice in prior works [1-3], we classify applications with zero detections as benign and those detected by at least four engines as malware. Following TESSERACT [2], we exclude grayware (applications detected by 1-3 engines), as assigning these ambiguous samples to either class could introduce labeling inconsistencies and disadvantage classifiers. We also release the raw VirusTotal detection scores, allowing future researchers to apply alternative labeling thresholds according to their evaluation requirements.

Fusion selection: In this work, ``best multimodal configuration” refers to the best performing evaluated fusion method. We do not focus on any hyperparameter fine-tuning because our goal is to benchmark baseline, off-the-shelf methods on McNdroid. We leave the development of optimized multimodal approaches to future researchers. We will consistently use the term 'best multimodal method' to prevent further ambiguity.

Dynamic features Interpretation: Our results demonstrate that multimodal fusion provides measurable benefits, with gains depending on the modality combination and temporal setting. As shown in Table 2, fusion improves FAR F1 from 35.7% to 47.9% for LightGBM and from 29.4% to 43.4% for Transformer. Dynamic features also contribute complementary information. For example, Table 7 shows that Static+Dynamic improves FAR SVM F1 from 33.3% to 41.6%. These findings reveal an important research direction in multimodal malware detection. Different modalities capture distinct aspects of malicious applications which allow one modality to potentially detect threats missed by another. Our cross-modal disagreement analysis in Section 6.3 also shows these differences. Instead of assuming that adding all modalities always improves performance, McNdroid allows researchers to investigate when modalities complement or conflict with each other and develop more effective fusion strategies for evolving malware threats.

Novelty: McNdroid is submitted under the ``Datasets and Benchmarks” category. Our primary contribution is a multimodal longitudinal benchmark of nearly 859K Android applications across 12 years, with aligned static, dynamic, and graph-based representations, along with binary and malware-family labels. Please note that there is not any multimodal malware benchmark at this scale. Cybersecurity and other relevant research community who study multimodal systems will be greatly benefited by McNdroid.

McNdroid provides pre-extracted features that support comparisons across modalities and future research on temporal drift, multimodal learning, and drift adaptation. Our contribution is the benchmark itself, we do not claim a new discovery about Android feature evolution and/or a new learning method.

Feature Extraction: Across all three modalities (including dynamic), we use the 2013 training partition as the reference for feature construction to avoid temporal leakage. We will clarify this consistent protocol in the revised manuscript.

References are in the next comment.

Please let us know if you have any further concerns, we will do our best to address those as well.

AuthorsOpenReview ↗

References

References:

[1] Haque, Md Ahsanul, et al. "LAMDA: A longitudinal android malware benchmark for concept drift analysis." International Conference on Learning Representations. 2026.

[2] Pendlebury, Feargus, et al. "TESSERACT: Eliminating experimental bias in malware classification across space and time." USENIX Security Symposium. 2019.

[3] Miller, Brad, et al. "Reviewer integration and performance measurement for malware detection." International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. 2016.

[4] Anderson, Hyrum S., and Phil Roth. "EMBER: An open dataset for training static pe malware machine learning models." arXiv preprint arXiv:1804.04637. 2018.

[5] Joyce, Robert J., et al. "EMBER2024-a benchmark dataset for holistic evaluation of malware classifiers." ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2025.

[6] AndroZoo. "AndroZoo API Documentation." https://androzoo.uni.lu/api_doc (accessed November 09, 2026).

[7] Chow, Theo, et al. "Beyond the tesseract: Trustworthy dataset curation for sound evaluations of android malware classifiers." IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). 2026.

[8] VirusTotal. "In-house Sandboxes - behavioural analysis products". https://docs.virustotal.com/docs/in-house-sandboxes.

AuthorsOpenReview ↗

Follow-up on Our Responses and Revisions

We have done our best to address your concerns in our responses and revisions. Before the discussion period ends, please let us know if any concerns remain or if further clarification would be helpful.

Comments on the paper

1 comment

Dear PC/SAC/AC/Reviewers,

We have carefully addressed all concerns raised by the reviewers and incorporated the corresponding changes into our revised manuscript. Below, we provide a summary of each reviewer's concerns alongside the resolutions taken in our updated rebuttal. We have also updated the paper pdf with addressed concerns.

We hope our responses help the reviewers, PC, SAC, and AC to support our paper.

Reviewer concernResolutionManuscript / Rebuttal Reference
Fusion selectionWe clarify the original fusion method selection procedure. We add an additional experiment using validation-only selection within the 2013 training partition, with configurations fixed across future years. Fusion improves F1 in over 70% cases compared to unimodal approaches.Revised: Sections 5 and 8\color{blue}{\textbf{Sections 5 and 8}}. Additional experiment: Nov. 16 response to MWhC, Part 1.
Class prevalenceWe include class-balanced evaluation. Similar temporal degradation persists under fixed class ratios which supports our claim that prevalence changes alone do not explain the decline. We also clarify family coverage and the ability to construct deployment specific class distributions.Added: Appendix F.2, Table 9\color{blue}{\textbf{Appendix F.2, Table 9}}. Clarified: Sections 5 and 8\color{blue}{\textbf{Sections 5 and 8}}; responses to MWhC and HCKt.
Data splits and statistical reportingWe clarify annual 80/20 partitions, 2013 training, and evaluation on subsequent yearly test partitions. We also include per year results.Revised: Section 3.1\color{blue}{\textbf{Section 3.1}}. Added: Appendix F.1, Figure 9\color{blue}{\textbf{Appendix F.1, Figure 9}}.
Multimodal research valueWe include APK-level error analysis. For example, LightGBM in 2024, graph-based detection recovers 42.70% of malware missed by static detection, with 0.19% FPR. We investigate shared failures, complementary detections, and false-positive costs to explain when additional modalities help and when fusion remains challenging.Additional analysis: Nov. 15 response to HCKt, Parts 1–3. Supporting analyses: Sections 5–7\color{blue}{\textbf{Sections 5–7}}; Appendix J\color{blue}{\textbf{Appendix J}}.
Feature construction and stability comparisonsWe clarify source-year fitting, explain how fixed vocabularies and hashing handle unseen attributes, and specify the representation dependent scope of stability rankings.Revised: Section 3.2\color{blue}{\textbf{Section 3.2}}. Detailed explanation: Nov. 16 response to MWhC, Part 1.
Timestamps and 2015 coverageClarified the use of AndroZoo's added timestamp and the absence of 2015 samples in our collection under this criterion.Revised: Section 3.1\color{blue}{\textbf{Section 3.1}}.
VirusTotal labeling and label changesWe clarify thresholds and labeling protocol. We also release original and updated detection counts and engine verdicts. We include the year wise benign to malware and malware to benign transitions(label drift).Revised: Section 3.1\color{blue}{\textbf{Section 3.1}}. Additional analysis: Nov. 16 response to MWhC, Part 2.
Dynamic feature coverage and usabilityWe extend the extraction description and add behavioral coverage statistics. We explain sandbox coverage limitations and that the released features can be used without new VirusTotal queries.Revised: Sections 3.2 and 8\color{blue}{\textbf{Sections 3.2 and 8}}. Clarifications: responses to HCKt, fa35, and MWhC.
Complete-modality filteringWe clarify that retaining the same applications across modalities enables controlled comparisons and acknowledges potential selection bias from extraction failures. We already examine sensitivity to missing modalities using the retained applications.Revised: Sections 3.1 and 8\color{blue}{\textbf{Sections 3.1 and 8}}. Existing experiments: Appendix J\color{blue}{\textbf{Appendix J}}.
Baseline coverage and contributionWe explain the choice of baseline models for comparability with prior longitudinal benchmarks. Our evaluation also includes HCC, CADE, and DAN-MMD for drift adaptation, alongside continual-learning methods and GNN baselines.Existing evaluations: Section 7\color{blue}{\textbf{Section 7}}; Appendices L, N–O\color{blue}{\textbf{Appendices L, N–O}}. Clarifications: responses to HCKt and MWhC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.