acceptodds
Under review as a conference paper at ICLR 2027

Are Datasets Reliable? Pitfalls in Evaluating Decomposition-Attack Detection

Abstract

Decomposition attacks against large language models (LLMs) bypass safety measures by splitting a malicious query into multiple sub-queries that appear individually benign. Evaluating and defending against such attacks requires datasets of realistic attack instances. However, existing decomposition attack datasets suffer from several limitations, one of the most critical being that many of the generated sub-query sequences could plausibly arise from a benign query, making them insufficient as genuine attack instances. To address this limitation, we propose the Intent Ambiguity Score (IAS), a metric that quantifies how plausible it is that a given decomposed sequence could also arise from decomposing some benign query. IAS is computed by predicting a benign query whose decomposition is semantically equivalent to a malicious query's sub-sequence, generating that task's own sub-sequence, and then cross-evaluating whether each sub-sequence is a natural decomposition of the other's original query. We formally establish the validity of IAS and compare it against human and LLM-based annotations. We also develop a new decomposition attack dataset generation pipeline that incorporates an IAS-based filtering mechanism. Our dataset is of high quality, achieving a strong attack success rate against closed- and open-weight models while evading existing defense mechanisms.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmark Dataset, Query Decomposition Attack, LLM Jailbreak

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_anXKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies key limitations in existing datasets for evaluating LLM decomposition attacks, including intent ambiguity, explicit malicious intent in individual sub-queries, and lexical differences between malicious and benign queries. To address these issues, the authors propose the Intent Ambiguity Score (IAS), which estimates whether a decomposed attack sequence could plausibly originate from a benign query, and develop an IAS-based dataset generation pipeline with a feedback loop for rephrasing suspicious sub-queries. Experiments across five LLMs show that the generated dataset achieves higher attack success rates than DecomposedHarm in both multi-turn and multi-session settings.

Critical strengths
  • The authors identify key flaws in existing decomposition-attack datasets, including intent ambiguity and lexical bias.

  • The authors propose an IAS-based dataset-generation pipeline with empirical validation against human judgments.

  • The experimental results demonstrate that the dataset generated by the IAS-based pipeline achieves attack success rates over DecomposedHarm across five LLMs and two evaluation settings.

Critical weaknesses
  • Significant lexical bias remains between malicious and benign queries, with a TF-IDF classifier achieving a ROC-AUC of 0.9963, raising concerns about whether the dataset reliably evaluates attack detection beyond superficial lexical differences.

  • The evaluation does not sufficiently establish that the proposed dataset improves the generalizability and reliability of attack-detection benchmarks, as the main evidence focuses on attack success rates across a limited set of models and settings.

Questions for authors

It would be better to discuss the paper’s limitations and future work.

3 comments

Thank you for reviewing our work. We address your comments below:

Significant lexical bias remains between malicious and benign queries, with a TF-IDF classifier achieving a ROC-AUC of 0.9963, raising concerns about whether the dataset reliably evaluates attack detection beyond superficial lexical differences. It would be better to discuss the paper’s limitations and future work.

We added Section 6 (Limitations) and discuss there that classifiers relying on lexical features may achieve overestimated performance on our dataset.

The evaluation does not sufficiently establish that the proposed dataset improves the generalizability and reliability of attack-detection benchmarks, as the main evidence focuses on attack success rates across a limited set of models and settings.

We agree that our evaluation is limited to a restricted set of models and settings. In the revised version, we narrowed our claim to providing ground truth in which the attack succeeds (Abstract, Section 1), and added a reconstruction-based verification as supporting evidence (Section 5.2.1).

Thank you for addressing my concerns.

Thank you for your response. We appreciate your valuable feedback, which helped us improve our work.

Reviewer_FRKnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper examines quality issues in datasets for decomposition attacks against LLM safety mechanisms, identifying three problems in existing benchmarks: intent ambiguity, malicious intent exposed in individual sub-queries, and strong lexical/topic bias between benign and malicious examples. To address these issues, the paper proposes the Intent Ambiguity Score (IAS) and an IAS-based dataset construction pipeline, evaluates IAS against human and LLM annotations, and compares the resulting dataset with DecomposedHarm across several models and defense settings.

Critical strengths
  1. The paper identifies an important benchmark-quality issue. A simple TF-IDF classifier can distinguish malicious and benign decompositions with very high accuracy, revealing strong lexical shortcuts in DecomposedHarm.

  2. The experiments cover multiple closed- and open-weight models and several attack/defense settings. The paper also clearly reports that lexical bias remains largely unresolved in the proposed dataset.

Critical weaknesses
  1. The attack-success metric is too weak. Treating non-refusal as success does not show that the combined sub-query responses are actually sufficient to answer the original malicious query. This weakens the main empirical claim.
  2. IAS is not validated strongly enough. It depends on several LLM-based steps, while the human study is limited and the error analysis reveals failures in both benign-query generation and naturalness judgment. A related limitation is that the lexical-bias issue remains almost unchanged in the new dataset.
Questions for authors
  1. Can the authors evaluate end-to-end attack success by checking whether the combined sub-query responses are sufficient to answer the original malicious query?
  2. How sensitive is IAS to the choice of predictor/evaluator model, the number of sampled benign queries, and the filtering threshold?
  3. The formal analysis assumes a deterministic decomposition mapping, whereas the actual LLM-based decomposition process is stochastic. Since the paper notes that the formal argument does not directly apply in this setting, could the authors clarify how the formal result relates to the actual IAS procedure?
3 comments

Thank you for reviewing our work. We address your comments below:

  1. The attack-success metric is too weak.
  2. Can the authors evaluate end-to-end attack success by checking whether the combined sub-query responses are sufficient to answer the original malicious query?

Following your comment, we added Section 5.2.1 in the revised version: a jailbroken model answers all sub-queries, and we confirm with StrongREJECT-based evaluation that an answer to the original malicious query can be reconstructed from these responses (Table 2). Building on this, Section 5.2.2 now explicitly reports a "refusal-based ASR." We discuss the difficulty of evaluating the true ASR in Section 6 (Limitations).

  1. IAS is not validated strongly enough.

We agree that IAS relies on multiple LLM-based steps and that the human study is limited in scale, so its current validation is not sufficient. We also recognize the failures in benign-query prediction and naturalness judgment revealed by the error analysis as directions for improving IAS. In the revised version, we toned down the positioning of IAS to a "useful signal" (Section 4.3) and added a sensitivity analysis with respect to the number of predictions N and the threshold (Appendix B.2). The remaining lexical bias is discussed in Section 6 as a limitation: classifiers relying on lexical features may achieve overestimated performance on our dataset.

  1. How sensitive is IAS to the choice of predictor/evaluator model, the number of sampled benign queries, and the filtering threshold?

We analyzed the sensitivity to the number of samples N and the threshold in Appendix B.2 of the revised version. The correlation nearly plateaus at N = 3, and it increases monotonically with the threshold up to 0.75 and remains nearly flat afterward. We have not yet examined the sensitivity to the choice of predictor and evaluator models, but we plan to add these results if they can be completed during the discussion period.

  1. The formal analysis assumes a deterministic decomposition mapping, whereas the actual LLM-based decomposition process is stochastic.

In Section 4.2 of the revised version, we re-model decomposition as a stochastic process. We also state explicitly in Section 4.2 how the formal analysis relates to the actual procedure: the formal analysis motivates the search strategy of IAS, while the naturalness score used in practice approximates semantic equivalence, and its validity is examined empirically in Section 4.3.

The revision addresses my main concerns about soundness, especially the reconstruction evaluation and stochastic formalization. I therefore raise my rating from 2 to 3. Good luck!

Thank you for raising your score! We appreciate your valuable feedback for improving our work.

Reviewer_sMDbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether the datasets used to evaluate decomposition-attack detection are reliable, and finds systematic quality problems in existing datasets (DecomposedHarm in particular): many sub-query sequences labeled as attacks could equally have come from benign queries. The paper proposes the Intent Ambiguity Score (IAS) and a data-generation pipeline with IAS filtering and a rephrasing feedback loop, defines IAS as a measure of how strongly a decomposition is tied to benign queries, proves Lemma 1 establishing the formal plausibility of IAS under Assumption 1 (origin space is invariant under semantic equivalence), and conducts experiments on 5 target models under multi-turn and multi-session settings, demonstrating that the new dataset's ASR is consistently higher than DecomposedHarm's.

Critical strengths
  1. All three problems are quantified, not merely exemplified: Section 3.2 reports that 39.5% of test decompositions contain a sub-query that is malicious on its own; Section 3.3 obtains ROC-AUC 0.9998 and macro F1 0.991 with TF-IDF + logistic regression. Every problem is backed by an in-paper measurement, rather than only qualitative examples.
  2. The IAS algorithm is fully described, and the bidirectional design is theoretically motivated: Algorithm 1 explicitly spells out the scoring formula and the max operation; the bidirectional naturalness examples in Fig. 2–3 clearly explain why queries are swapped rather than directly asking an LLM to judge semantic equivalence.
  3. The new dataset consistently outperforms DecomposedHarm in ASR across multiple settings: All 10 conditions (5 models × 2 conversational settings) show improvements, with the largest gains on GPT-5. ASR also leads substantially on 7B open-source models, indicating that the dataset quality improvement generalizes across models.
Critical weaknesses
  1. The claim "evading existing defense mechanisms" does not fully match Table 4: The Abstract and Section 6 state that the new dataset can "evade existing defenses," but Table 4 shows that while the Single-Query Judge's ASR does rise from 0.605 to 0.723, the Incremental Judge's ASR actually falls from 0.216 to 0.114, suggesting that incremental monitoring is even better at catching the new dataset's decompositions. The authors explain the drop as "malicious intent only emerges when the pieces are combined," but this explanation is in tension with the literal meaning of "evading".
  2. The ASR evaluation criterion does not match the attack definition in Section 1: Section 1 defines the attack as "aggregating the responses to the sub-queries to reconstruct the answer to the original malicious question," but the success criterion in Section 5.2.1 is: failure if any sub-query's response contains a refusal string, otherwise success. The paper does not check whether the combined responses actually constitute a harmful answer, so ASR may overestimate the real attack effectiveness.
  3. The claim "formally establish the validity of IAS" is too strong: Lemma 1 only proves a sufficient condition under Assumption 1, and relies on assuming that decompose is a deterministic mapping; the closing sentence of the same section explicitly admits that real decomposition is a stochastic LLM process, so the argument does not carry over as is. There is no theorem connecting the naturalness scoring in Algorithm 1 to the O(s) in the formal definition.
Questions for authors
  1. Does the new dataset's ASR advantage come mainly from rephrasing or from IAS filtering? The improvement in Section 5.2.1 is attributed to the rephrasing feedback loop, but the before/after filtering comparison in Section 5.2.2 (mean IAS 0.758→0.524) keeps only 36.4% of the data. Can the independent contributions of these two factors be separated?
  2. What is the basis for the filtering threshold 0.75 and N=3? Appendix C explains that 0.75 is the threshold with the highest correlation with human judgment, but no threshold sweep table is given. Is N=3 predicted queries enough to cover the high-dimensional origin space?
  3. In the appendix, GPT-5 + ICL has a higher Spearman (0.729) than IAS (0.606); why not use GPT-5 + ICL directly for filtering? The authors say commercial models may refuse the task, but the refusal rate is not quantified. If GPT-5 + ICL were available, what would IAS's advantage be?
2 comments

Thank you for reviewing our work. We address your comments below:

  1. The claim "evading existing defense mechanisms" does not fully match Table 4

We agree that "evading existing defense mechanisms" was an overstatement. The goal of our dataset is not to bypass defenses but to provide ground truth in which the attack succeeds once all sub-queries are answered. Since attackers are free to choose how the sub-queries are actually presented (e.g., obfuscation or injecting them in a very long context), evading defenses is not the aim of this work. We have therefore removed this phrase from the Abstract, the contributions in Section 1, and the Conclusion (Section 7), and adjusted the wording accordingly.

  1. The ASR evaluation criterion does not match the attack definition in Section 1

We agree that the original version did not verify whether the combined responses yield a harmful answer. In the revised version, we added Section 5.2.1: a jailbroken model answers all sub-queries, and we confirm with StrongREJECT-based evaluation that an answer to the original malicious query can be reconstructed from these responses (Table 2). Building on this, Section 5.2.2 now explicitly reports a "refusal-based ASR." Note that evaluating the true ASR is inherently difficult: LLM judges have limited reliability, and human evaluation requires domain expertise in each harm category. We also discuss this in Section 6 (Limitations).

  1. The claim "formally establish the validity of IAS" is too strong

We agree that "formally establish the validity of IAS" was too strong. In the revised version, we reposition the formal analysis as providing a formal motivation for the search strategy of IAS, and we re-model decomposition as a stochastic process (Section 4.2). We also state explicitly in Section 4.2 that the naturalness score is an approximation of semantic equivalence, whose validity is examined empirically in Section 4.3.

  1. Does the new dataset's ASR advantage come mainly from rephrasing or from IAS filtering?

We have not yet conducted an ablation that separates the two contributions. Since IAS filtering retains only decompositions that preserve the malicious goal more faithfully, filtering alone would be expected to lower ASR. The fact that ASR nevertheless improves suggests that the gain comes mainly from rephrasing (Section 5.2.2). We plan to add this ablation if time permits during the discussion period.

  1. What is the basis for the filtering threshold 0.75 and N=3?

We added a sensitivity analysis in Appendix B.2 of the revised version. The correlation with human annotations nearly plateaus at N = 3. For the threshold, the correlation increases monotonically up to 0.75 and remains nearly flat afterward. We chose these values accordingly

  1. why not use GPT-5 + ICL directly for filtering?

The advantage of IAS is that it works with a lightweight open-weight model. Although we have not quantified the refusal rate of commercial models, retrying until the model does not refuse and using only the non-refused judgments could bias the results toward cases in which the model is willing to respond. In addition, as shown in the error analysis in Appendix B.1, the main sources of divergence between IAS and human annotations are errors in benign-query prediction and naturalness judgment, both of which can be reduced by using a more capable model. We therefore expect the accuracy of IAS to improve as open-weight models become more capable.

Following up on Q1, we have added an evaluation of the effect of IAS filtering on ASR in Appendix B.3. The results show that the ASR improvement stems mainly from our decomposition process and the rephrasing feedback loop rather than from IAS filtering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.