- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies adversarial evasion against malware detectors that use large language models (LLMs) to reason over textual renderings of program features. It argues that converting attacker-influenced APK features into natural language creates attack channels beyond conventional byte- or feature-space manipulation. The paper introduces LLM-DE, a framework organized around four attack categories: prompt injection through feature descriptions (PIPR-A), exploitation of safety-alignment and refusal behavior (SAE-A), parsing-error and truncation induction (PEI-A), and semantic camouflage using benign application signatures (SC-A). It further proposes mechanism-driven optimized variants, including safety inversion, two-sided semantic neutralization, targeted truncation, and a unified composition, together with informal analyses of submodular feature selection, stealth, and asymmetric transfer to classical detectors.
Validation uses a synthetic 215-dimensional DREBIN-style Android feature corpus, a fixed feature-to-text converter and system prompt, six instruction-tuned LLM backbones, and two classical detectors. The measured end-to-end LLM results are primarily obtained on Phi-3-mini using only three malicious samples; Qwen2.5-0.5B/1.5B are shown to produce input-invariant outputs, while results for Qwen2.5-7B, Llama-3.1-8B, and a GPT-4-class model are pilot-anchored projections rather than measurements. On Phi-3-mini, PIPR-A, SAE-A+, SC-A+, and Unified+ flip all three measured malicious samples, whereas PEI-A+ does not. Feature additions also transfer strongly to a linear DREBIN-SVM, with up to 99.7% ESR for PIPR-A and 79.1% for Unified+. The paper concludes that LLM-based malware detection remains vulnerable to both text-level and feature-level adversarial manipulation.
Critical strengths
-
The paper identifies a timely and underexplored security problem. Rather than treating an LLM detector as an ordinary classifier, it analyzes the full feature-to-text-to-model pipeline and makes explicit how attacker-controlled representations can become instruction channels. The four-way taxonomy, covering prompt injection, alignment exploitation, parsing/truncation, and semantic camouflage, provides a useful vocabulary for auditing LLM-based security systems beyond malware detection. This framing is consequential because it connects malware evasion with broader prompt-injection and agent-security concerns, and it suggests concrete defensive requirements such as separating trusted instructions from untrusted feature text. Although the current validation is insufficient for strong acceptance, the problem formulation and taxonomy could motivate substantial follow-up work.
-
The mechanism-driven analysis produces informative, testable hypotheses rather than reporting only attack success rates. The paper explicitly investigates why naive SAE-A, PEI-A, and SC-A fail on Phi-3-mini, then proposes targeted variants based on refusal intent, two-sided evidence control, and deterministic truncation. The resulting measured changes, including 0/3 to 3/3 flips for SAE-A+ and SC-A+, provide a useful proof-of-concept that analyzing model behavior can guide adversarial construction. The study also reports negative results, per-sample probabilities, baseline comparisons, and transfer asymmetry between a linear SVM and an MLP. These details improve interpretability and reproducibility, even though they do not overcome the paper’s limited sample size and synthetic evaluation setting.
Critical weaknesses
-
The central empirical claims rest on an extremely small and narrow evaluation. The decisive end-to-end results are obtained with one discriminating backbone, Phi-3-mini, and only three malicious samples. The other small models are input-invariant and therefore do not constitute meaningful detector evaluations. A three-sample sweep cannot establish robustness, reliability, or generalization across malware families, feature combinations, prompts, or random seeds; a 3/3 success rate can easily reflect sample selection or a pipeline-specific artifact. The paper also uses a synthetic feature corpus rather than executable, repackaged APKs, so it does not demonstrate that the proposed additions are realizable while preserving malicious behavior. Because the paper’s strongest conclusions concern systematic vulnerability and broad attack-surface coverage, this limited evidence is consequential and insufficient for an ICLR-level empirical security claim.
-
The headline cross-backbone results are projections rather than measurements. The table that supports claims about Qwen2.5-7B, Llama-3.1-8B, and GPT-4-class detectors is generated by a pilot-anchored model using assumed capability and channel-susceptibility parameters. These estimates are explicitly not experimental outcomes, yet they are presented alongside measured ESR values and are repeatedly used to argue that attacks strengthen with model scale and alignment. Published jailbreak priors do not validate this particular malware-detection pipeline, and the model’s independence assumptions are untested. The measured Phi-3-mini outcomes even contradict some projected behavior for the naive attacks. Consequently, the paper does not currently establish transfer across backbones or the claimed near-universal effectiveness; it establishes only a promising hypothesis that requires direct experiments on the advertised target models.
-
The threat model does not convincingly establish an executable payload path. The framework assumes that adding benign permissions, APIs, or intents can cause the feature-to-text converter to emit arbitrary instruction payloads, but the converter is otherwise described as deterministic and based on canonical feature descriptions. If the payload instead comes from app labels, library descriptors, or string resources, the paper must specify which attacker-controlled fields are parsed, how they survive APK rebuilding, and whether the deployed detector actually includes them in its LLM input. Merely adding a feature does not generally let an attacker rewrite its canonical description. This unresolved mismatch affects the feasibility of PIPR-A and the unified attacks, and it also undermines the “functionality-preserving” claim because no real APK construction, installation, execution, or behavioral equivalence test is provided.
Questions for authors
-
Clarification on measured versus projected evidence: Which numerical results in the main paper were obtained from direct end-to-end queries to each named backbone, and which were generated by the pilot-anchored projection model? In particular, can the authors confirm that no direct measurements support the Qwen2.5-7B, Llama-3.1-8B, and GPT-4-class ESR values in Table 1, and explain why these projections are used to support cross-backbone conclusions?
-
Clarification on functionality preservation: Were any attack samples instantiated as valid APKs and checked for successful rebuilding, installation, execution, and preservation of the original malicious behavior? If not, what precisely does the paper mean by “functionality-preserving” beyond the feature-addition-only constraint, and which reported attacks rely on modifications that are not represented in an executable artifact?
-
Clarification on the ESR success criterion: How many reported successes are actual MALICIOUS-to-BENIGN flips, versus refusals or unparseable outputs? Would the main conclusions, especially for SAE-A, remain unchanged if refusal and malformed responses were mapped to MALICIOUS under a fail-closed detector policy? Please also clarify whether the reported first-token verbalizer mass was calibrated or validated against the detector’s final classification decision.
Direct-backend inventory, fail-closed ESR, and functionality-preservation correction
Thank you for the focused clarification questions.
Which results are direct? Direct end-to-end attack measurements exist only for Phi-3-mini on three malicious samples, with five benign clean controls. PIPR-A, SAE-A+, SC-A+, Unified, Unified+, B2, and B3 were directly queried in their respective batches; PEI-A+ did not flip the three samples. The released naive-Unified CSV contains only two rows although Table 15 reports three, so we will not use the missing third row as auditable evidence. The small Qwen2.5-0.5B/1.5B results are collapse/probe observations, not useful attack evaluations. We confirm that no direct measurements support the Qwen2.5-7B, Llama-3.1-8B, or GPT-4-class ESR values in Table 1. Those values, and the capability-scaled Phi-3 entries, come from the projection model. Classical SVM ESR, assumed channel susceptibility, and probe accuracy have no validated mapping to LLM ESR. We will remove the projected tables and stop using them for cross-backbone or capability conclusions.
What does functionality preservation mean here? No attack sample was instantiated as an APK, and no rebuilding, installation, execution, or malicious-behavior test was performed. In the submitted code it means only that a synthetic binary feature is not removed. All attacks operate on synthetic vectors; PIPR-A and unified variants additionally rely on a payload directly appended by our renderer. We have not demonstrated an attacker-controlled APK field that survives an unchanged extractor/converter into that position. Adding a feature cannot itself rewrite a canonical description. We will therefore remove “functionality-preserving” as an executable-artifact claim and narrow the threat model to the constructed feature-to-text pipeline.
How are successes counted? The original CSVs contain only a top-20 first-token score, not final generated responses. The scorer aggregated B*/M* token prefixes and renormalized over their sum; this was not calibrated against complete generations, was not a complete class-label likelihood, and returned 0.5/0.5 when class tokens were absent. We therefore reran all 26 original cases and saved final generations. Exact whole-class token mass over the full first-token vocabulary was zero in all 26 cases because both labels span multiple tokens. Final generations produced no REFUSAL or semantically unparseable outputs, but two lacked the required prefix. For the three clean-detected malicious samples, explicit BENIGN counts were: clean 0/3; PIPR-A 3/3; B2 3/3; SAE-A+ 3/3; SC-A+ 3/3; PEI-A+ 0/3; Unified+ 3/3. Two outputs used bare first-line
BENIGNinstead ofCLASSIFICATION: BENIGN. Under strict-format fail-closed scoring they are malformed and blocked, so PIPR-A and Unified+ become 2/3; other counts are unchanged. Thus semantic fail-closed counts are unchanged, while strict formatting reduces two attacks. Complete-label sequence scoring on nine clean cases plus one PIPR-A case disagreed with generation for 2/5 benign controls, so this limited audit is not a calibration validation. All raw responses are preserved.For fail-closed ESR, we condition on the three originally detected malicious samples and count only an explicit generated BENIGN as an escape. MALICIOUS, REFUSAL, and OTHER all remain blocked. This is the appropriate policy when uncertain output is quarantined rather than admitted. The new diagnostic audits measurement on the same samples; it does not remedy the tiny sample, establish calibration on a representative dataset, or demonstrate APK realizability.
We agree that the supported conclusion is only a limited proof of concept on this pipeline. Broad claims of systematic vulnerability, cross-backbone transfer, and practical malware evasion require direct large-scale backbone tests and executable APK/system-path validation.