acceptodds
Withdrawn from ICLR 2027

Which Pairs Should Agree? Triangular Consistency for Adversarial Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Consistency regularization has become a standard component of adversarial train- ing (AT), encouraging agreement between predictions on related inputs. We ar- gue that existing formulations are structurally incomplete. Given two augmented views of the same sample, each subjected to an adversarial perturbation, there are two distinct consistency constraints: one connecting each clean view to its corresponding adversarial example, and another connecting the adversarial exam- ples across views. Existing objectives enforce only the former constraint, leav- ing part of the predictive geometry unconstrained. We jointly impose both con- straints, forming a triangular structure among the predictive distributions, which we term triangular consistency (TC). We provide a formal analysis showing why this combination is important. Specifically, the consistency terms contribute to a Rademacher-based bound on the robust generalization gap, while a triangle- inequality argument demonstrates that bounding the divergence between a clean prediction and an adversarial prediction from the other view requires both con- straints. Empirically, TC achieves the strongest robustness among the evaluated methods under ℓ∞ attacks. In addition, a term-count-matched ablation indicates that the improvement stems from the structural arrangement of the consistency constraints rather than simply from introducing additional loss terms. Finally, the robustness induced by TC transfers beyond the training setting, improving perfor- mance under larger unseen perturbation budgets as well as non-ℓp attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, consistency regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8rqAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Triangular Consistency (TC), a consistency-regularization method for adversarial training. The main idea is to jointly enforce within-branch consistency between each clean view and its corresponding adversarial example, and cross-branch consistency between adversarial examples generated from different augmented views. The resulting objective combines three Jensen--Shannon divergence terms, Lc1L_{c1}, Lc2L_{c2}, and Lc3L_{c3}. The authors provide theoretical analyses involving robust generalization, prediction stability, and local curvature. Experiments on CIFAR-10 and CIFAR-100 show improvements over several adversarial training baselines under PGD attacks, with additional results on larger perturbation budgets and non-ℓp\ell_p attacks.

Critical strengths
  1. The proposed triangular consistency objective is intuitive and easy to integrate into existing adversarial training frameworks without additional network modules. Jointly considering within-branch and cross-branch consistency is a reasonable direction.

  2. The experiments cover different architectures and attack methods on CIFAR-10 and CIFAR-100. TC achieves improvements under PGD and CW attacks, while the results on StAdv suggest potential benefits beyond the training threat model.

Critical weaknesses
  1. The proof of Theorem 1 in Appendix C appears to contain inconsistencies. Under the stated construction, f1f_1 misclassifies one adversarially perturbed sample, suggesting that its robust training error should be 1/21/2 rather than zero. Furthermore, the subsequently introduced classifier f3f_3 does not appear to achieve zero triangular consistency loss under the specified perturbations. These issues raise concerns about the correctness of the theoretical analysis.

  2. The role of Lc2L_{c2} in the theoretical analysis is unclear. In Eq. (11), the triangular consistency defect only involves Lc1L_{c1} and Lc3L_{c3}, and the triangle inequality in Eq. (13) also relies solely on these two terms. This suggests that the theoretical argument may only require Lc1+Lc3L_{c1}+L_{c3} rather than the full three-edge objective. However, Table 5 does not include this particular two-edge combination, making it difficult to assess whether all three constraints are necessary.

Questions for authors

Q1. Could the authors revisit the proof of Theorem 1? Additionally, could the authors clarify how Eq. (49) in Proposition 3 follows from the Lipschitz assumption? Are further assumptions required?

Q2. In Table 5, the full TC objective improves PGD accuracy but performs worse than the two-edge objective under AutoAttack on both datasets. How should this result be interpreted? Could the authors also provide a comparison in which the effective regularization strengths of the cross-branch and decoy terms are better matched?

Reviewer_e7jjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves consistency regularization by triangular consistency, which enforces consistency among the input and two adversarial examples. The paper notes existing approaches that impose consistency on input-adversarial pairs and on adversarial-adversarial pairs, then tries to combine them to take the benefits of both. The proposed method applies two different label-preserving transformations to a given input, and generates two different adversarial samples from them. The paper defines three additional loss terms using Jensen-Shannon divergences between the two input-adversarial pairs and the adversarial-adversarial pair. Various theoretical analyses justify the benefit of triangular consistency. These analyses include bounding the generalization bound, bounding the probability of an adversarial event, and a geometric interpretation of the potential improvements. Experiments demonstrate the performance on standard and unseen attack settings. Ablation studies explore additional experimental questions: the necessity of all loss terms, the choice of divergences, and the weight on adversarial-adversarial pair consistency.

Critical strengths
  1. The idea has clear motivation and potential to improve the existing consistency regularization. The design seems to be naturally derived from combining previous approaches.
  2. The paper contains theoretical analyses from various perspectives. These analyses provide a solid theoretical foundation for the proposed method and improve the paper's overall soundness.
Critical weaknesses
  1. The paper needs clearer presentation of its findings. For example, Section 4 jumps between topics, and it is hard to extract the section's key takeaway. The section structure is poorly organized, and the paper spends space on marginal findings while omitting important information.
  2. While the proposed method improves robustness against the canonical threat model, it performs worse than competitors under non-ℓp\ell_p attacks. Without justification for this performance gap, these results undermine the practical value of the proposed method.
Questions for authors
  1. Please improve the overall presentation
    • Not all results are worth presenting in the main body of the paper. Please focus on a few findings to emphasize, and move the others to the Appendix. For example, I don't see such meaningful findings from testing different divergence choices, β\beta-sensitivity, or robustness against FGSM. Be more selective about what you present in the main body so you can move other important details to the main body of the paper.
    • Do not omit the essential parts by referring to the Appendix. The paper may not include the full related-works and experimental-setup sections in the main body, but it should describe all the attack and defense methods abbreviated in the results tables without referring to the Appendix.
    • Organize the section structure more clearly. I propose turning some paragraphs (starting with boldface phrases) into subsections and making Section 6 a subsection of Section 5.
    • At the beginning of Section 4, please summarize what theoretical analyses have been performed and the key takeaways from the theoretical analysis.
  2. Improve the quality of the experiments
    • The results contain many attack and defense methods. However, the experiments used only the CIFAR-variant dataset and ResNet-variant model. Please include more results on different datasets and model architectures to show the practical value of the proposed method.
    • From the experimental results and Appendix B, all the attack-generated ℓ∞\ell_\infty-perturbations. Appendix B clarifies that the paper uses only this attack type unless other types are stated explicitly, but I cannot find any other ℓp\ell_p-perturbation attack in the paper. A natural unseen attack is an ℓp\ell_p-norm attack where pp differs from the training value, and standard adversarial training also suffers from defending against those. It would be meaningful to include those types of attacks in the experiments.
    • Justify the lower performance in Tables 2 and 3. It seems the authors are overly lenient on these weak results, emphasizing only cases where the proposed method performed well. Depending on the reader, such lower performance could be a practical concern. Please provide some justification about what factors could explain such degradations or why it would not be a practical issue with the proposed methods.
  3. Minor issues
    • Some in-text citations should appear as parenthetical citations. I found two such cases (Section 1 and Appendix A), but there could be more.
    • In Assumption 1-(i), μ\mu is the marginal of π\pi instead of δ\delta? Please go through the formal notations and ensure there is no typo.
    • Section 5 starts with the questions to be answered from the experiments. Sections 5 and 6 answer those questions in the order Q1, Q3, then Q2. Swapping Q2 and Q3 would give a more natural question order that matches the order of the answers presented.
    • The experimental result tables report results on CONS, but I cannot find the method CONS refers to. At least, it is not described in the related work section. If this method is not described in the paper, please include the description.
    • In Appendix A, there is a strange question mark, possibly due to a missing reference.
Reviewer_hEeeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines which prediction pairs should be constrained by consistency regularization in adversarial training. Given two augmented views of the same input and their corresponding adversarial examples, the proposed triangular consistency (TC) objective combines two within-branch clean–adversarial Jensen–Shannon divergences with one cross-branch adversarial–adversarial Jensen–Shannon divergence. These regularizers are added to adversarial cross-entropy losses on the two augmented views. The paper attempts to justify the cross-branch constraint through an insufficiency construction, a generalization bound under an extended threat model, a curvature interpretation, and a local stability certificate. Experiments use CIFAR-10/100 with ResNet-18 and WideResNet-34-10, evaluating PGD, CW∞, AutoAttack, larger perturbation budgets, and several non-L_p attacks. The paper also includes term-count-matched ablations and three-seed results for TC.

Critical strengths

This paper provides substantial experimental results. In addition to attacks within the training threat model, the paper evaluates StAdv, Recolor, PPGD, LPA, and larger L_∞ perturbation budgets.

Critical weaknesses

1. Fundamental errors in the theoretical analysis.

The theoretical results contain several critical flaws that undermine the paper's main claims:

  • Theorem 1 (Appendix C): The constructed classifier f3f_3 is identical to f2f_2, and its triangular consistency loss is 2log⁡22\log 2, not zero as claimed. Therefore, the construction fails to establish the claimed minimizer.
  • Theorem 2 (Eqs. 33–37): The proof incorrectly extends a gradient bound from the training perturbation region to a larger region without sufficient assumptions. Equation (35) also relies on an unjustified approximation. Consequently, the proposed generalization bound is not established.
  • Proposition 3 (Eq. 49): The proof incorrectly assumes that prediction differences within an input-space convex hull can be bounded by convex combinations of prediction differences at its vertices. Lipschitz continuity alone does not justify this inequality, invalidating the claimed local robustness certificate.

These are substantive mathematical errors rather than minor presentation issues.

2. Insufficient novelty and contradictory experimental evidence.

  • Limited novelty: Tack et al. (2022), Consistency Regularization for Adversarial Robustness, already combine within-branch clean–adversarial consistency with cross-branch adversarial consistency in their extended TRADES/MART formulations. The claimed novelty of jointly imposing these constraints is therefore not adequately justified.
  • Contradictory ablation results: Table 5 shows that adding the third consistency edge decreases AutoAttack accuracy from 50.10% to 49.35% on CIFAR-10 and from 26.59% to 26.30% on CIFAR-100. This contradicts the claimed robustness improvement and the trend described in Figure 3.
  • Insufficient evidence: The experiments do not convincingly demonstrate that the triangular structure itself provides a consistent improvement over existing consistency regularization methods. Direct comparisons with the closest prior formulations are needed.

Together, these issues substantially weaken both the theoretical validity and the claimed methodological contribution of the paper.

Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.