acceptodds
Under review as a conference paper at ICLR 2027

Fixed Activation Bounds Are Not Enough: A Distortion-Matched Transformer Study

Abstract

There are increasing societal concerns regarding safety around AI. Safe AI requires accurate predictions that withstand adversarial perturbations. Fixed bounds on internal activations constrain representation magnitudes, but whether they improve decision robustness remains uncertain. Here we show that fixed, training-calibrated hard and smooth caps preserve clean accuracy without establishing the prespecified practical robustness benefit in binary DistilRoBERTa classification on the Internet Movie Database benchmark. Across 55 matched training runs, we compare five transformer locations while matching mean absolute intervention on training-only activations; every nontrivial hard-cap distortion admits a unique positive smooth scale. All ten capped configurations satisfy corrected clean balanced-accuracy equivalence within one percentage point. At the primary embedding-space budget, none meets robustness-benefit criteria. Among 27 endpoint contrasts for caps with recorded activity, 15 establish corrected equivalence and 12 remain unresolved; the historically inactive smooth-logit comparator adds three equivalence decisions. At the less saturated budget, a post-hoc paired analysis likewise finds no corrected positive cap-control difference: hard query/key capping is 5.56 percentage points worse, seven contrasts remain unresolved, and two are equivalent. A contextual FreeLB-style baseline confirms that the evaluator distinguishes a more robust trained system. Our findings show a clear boundary between fixed activation control and robustness gains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, transformer representations, activation capping, distortion matching, model calibration, practical equivalence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mAivOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether fixed bounds on internal activations improve the adversarial robustness of a transformer classifier. It compares a hard cap, which clips large activations, and a smooth tanh-based cap, applied at one of several locations in DistilRoBERTa (embeddings, query/key, attention output, MLP, and logits). The cap level is calibrated on the activations of a trained uncapped model and then frozen, and the smooth cap is matched to the hard cap so that both change the activations by the same average amount. Models are retrained with the caps and evaluated on IMDb against embedding-space PGD attacks, using multiple seeds, paired statistical tests, and equivalence testing. The caps preserve clean accuracy but do not improve robustness at any location, and some caps reduce robustness under a smaller attack budget. An adversarial training baseline is much more robust, which shows that the evaluation can detect a truly robust model.

Critical strengths
  1. The experimental and statistical design is careful: matched seeds, paired tests with multiplicity correction, equivalence testing instead of inferring "no effect" from non-significance, adaptive attacks with BPDA for hard caps, and a FreeLB-style positive control showing that the evaluation can detect a robust model.

  2. The paper reports all configurations and results honestly, including negative and unresolved outcomes, and the distortion-matching idea gives a fair way to compare hard and smooth operators at the same intervention strength.

Critical weaknesses
  1. The studied method has almost no effect, so the paper offers little useful information. Across all locations and both cap types, the caps neither help nor clearly hurt, and the results are mostly null or unresolved. This is partly built into the setup: under the main attack budget, almost every model, including the uncapped control, is broken on nearly all examples, so no method could show an improvement.

  2. The contribution is limited. The method is simple clipping, and the theory is basic: Theorem 1 follows directly from the fact that a continuous, decreasing function has a unique solution, and the other results are standard bounds. A negative result can be useful, but this one neither challenges a common belief nor explains why the caps fail.

  3. The scope is narrow. The study uses one small model, one binary sentiment dataset, and one cap level. The attack changes word embeddings directly rather than real text, and under a real word-substitution attack (TextFooler), capped and uncapped models behave almost the same. It is unclear whether the findings matter in practice.

  4. The paper is hard to read and hard to reproduce. Most of the text is spent on statistical and procedural caveats, which hide a simple main message. Only result files are released, without training or attack code, and the appendix reports code fixes made after accessing the test set and an excluded experiment.

Questions for authors
  1. What new insight does this negative result provide beyond prior work showing that bounded activations do not imply robustness, and can the authors explain why the caps fail?

  2. Do the conclusions hold for larger models, multiclass tasks, other datasets, or realistic text attacks?

  3. Will the authors release the training and attack code, and can the main text be shortened to focus on the core findings?

Reviewer_ZidPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper compares fixed hard and smooth activation caps at five locations in DistilRoBERTa. Their intervention strengths are matched on training activations. On IMDb, the caps preserve clean accuracy but do not establish a practical robustness gain under the tested settings.

Critical strengths

1.Matching intervention strength makes the hard/smooth comparison more meaningful than simply using the same threshold. 2.The statistical treatment is careful. The authors distinguish equivalence from inconclusive results and use paired model seeds for inference. 3.BPDA checks, stronger attacks, and the FreeLB-style comparator are useful additions. The paper also acknowledges limitations that could otherwise make the negative result misleading.

Critical weaknesses
  1. The primary experiment is close to an accuracy floor. At 0.05r, robust accuracy is only 0.68–1.94% across the primary configurations. This shows that none is effective at this budget, but reveals little about differences at smaller budgets. FreeLB demonstrates sensitivity to a large gain, rather than resolving this issue for modest cap effects.
  2. The lower-budget results remain inconclusive for most caps. At 0.025r, seven contrasts are unresolved. Hard attention-output capping, for example, improves accuracy by 1.62 percentage points, with a 99% interval of [-3.15, 6.39]. A useful improvement is therefore not ruled out. The broader claim of a “clear boundary” seems stronger than the evidence supports.
  3. Only one capping strength is tested: the 99.5th percentile. Without a threshold sweep, it is difficult to distinguish a limitation of fixed capping from a limitation of this calibration policy. Showing the clean–robust accuracy trade-off across several thresholds would strengthen the negative result.
  4. Distortion matching is performed before capped fine-tuning. Since the models are subsequently trained separately, their activation distributions and actual intervention strengths may diverge. Please report realized distortion on clean and attacked inputs to clarify how comparable the operators remain after training.
  5. The valid experiments cover only one model and one binary classification dataset. The matching theorem establishes a well-defined calibration procedure, but does not explain when capping fails. An additional model or task would strengthen the broader takeaway.
Questions for authors
  1. Do the conclusions hold across several clipping quantiles?
  2. Can stronger attacks also be evaluated at 0.025r, where the accuracy floor is less restrictive?
  3. Will the calibration and attack code be released? Numerical summaries alone are insufficient to reproduce the study.
Reviewer_d6EnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigated whether Fixed Activation Capping can improve the robustness of a Transformer-based classifier or not. The author compare the hard Cap and smooth Cap and propose the Mean Absolute Distortion Matching to control variable changes in the experiments. The experiments focused on DistilRoBERTa on the IMDb sentiment classification task and showed stable clean accuracy. According to the robustness improvement criterion by the authors, no cases showed improvement in robustness. The authors claim that controlling activation magnitudes alone is insufficient to establish more robust model decisions. Therefore, this paper can be regarded as research that presents negative results but with systematic experimental evaluation.

Critical strengths
  1. Comprehensive and Rigorous Experimental Evaluation: The paper provides very comprehensive experiments to evaluate the robustness improvement for the targeted model. The experiments cover different capping locations, randomized seeds, and different capping methods, and use an appropriate attack evaluation method to avoid the potential influence caused by gradient masking. These can directly support the authors’ claim about the relationship between the robustness improvement and activation magnitude controls.

  2. Logical and Rigorous Theoretical Analysis: The paper provides a clear and rigorous theoretical foundation for comparing the different capping methods. The authors prove that there is a unique positive smooth cap parameter corresponding to hard cap distortion. A series of theoretical analyses provides a rigorous foundation for comparing the different capping methods in the experiments.

Critical weaknesses
  1. Limited Novelty: The hard cap and smooth capping are existing activation-bounding operations; the Mean Absolute Distortion Matching proposed in the paper can only be regarded as a variable control method, and there is limited discussion about how different capping locations and methods produce different robustness behaviors. Therefore, the novelty of this paper mainly based on the soundness of its experimental instead of new understanding about the adversarial robustness.

  2. Limited Generalizability: Although the paper has comprehensive experiments on target models and binary classification tasks. Different transformer architectures, model sizes, and more complex classification tasks may exhibit different behavior. Moreover, the robustness evaluation method in the paper focuses on continuous perturbations in the embedding space instead of attacking the original text. The supplementary experiments are also based on representation-space attacks. Therefore, the understanding of activation magnitudes controlling gain from the paper's negative results can hardly be generalized.

Questions for authors

There is limited explanation about different robustness outcomes across capping locations and methods. For example, the hard cap at the Query/key location has a noticeable decrease, but the others show different outcomes. Could the authors further explain the reasons?

The experiments are mainly based on DistilRoBERTa and binary classification tasks. Could the authors provide extra experiments or analysis to show that the negative results can be generalized to other transformer architectures and multiclass classification tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.