acceptodds
Under review as a conference paper at ICLR 2027

Lipschitz-Constrained Selective State-Space Models with State-Retention and PAC-Bayesian Bounds Against Hidden-State Poisoning

Abstract

Selective state-space models (SSMs) such as Mamba are efficient linear-time alternatives to attention, but their input-dependent discretisation creates an architecture-specific attack surface: short adversarial token sequences can drive the step size Δt toward saturation so that ¯At = exp(ΔtA) → 0, erasing prior context (hidden-state poisoning, HiSPA). Existing defences are detectors that offer no analytical guarantee. We introduce LipMamba, a selective-SSM parameterisation with spectrally-normalised projections, an eigenvalue-bounded state matrix and a discretisation step clamped to [Δmin,Δmax], and prove three results with complete proofs: (i) an explicit Lipschitz constant for the selective (input-dependent) scan on the bounded-input domain, which identifies the lower clamp Δmin > 0 as the condition that keeps the state bounded and thereby makes precise the error explosion previously observed in input-dependent SSMs; (ii) a state retention bound giving an analytically computed trigger length ℓ⋆ below which no trigger can reduce the hidden-state norm below a chosen fraction of its pre-trigger value, which removes the ¯At → 0 mechanism of HiSPA; and (iii) a PAC-Bayesian bound on adversarial risk whose gap term is governed by the same constant. We are explicit about scope: the analytical global constant grows geometrically with depth and yields a vacuous global certificate, so per-input radii are reported under a local Lipschitz estimate and are empirical; the retention bound constrains norm, not content, and for the constraint values used here it certifies triggers of one token at 50% retention and of up to six tokens against near-complete erasure. On RoBench-25 and HarmBench, LipMamba-370M raises poisoned accuracy under HiSPA at trigger length 24 from 23.4% (Mamba) to 85.3%, retains 80.4% accuracy at empirical local-Lipschitz radius 0.18, and adds 3.5–4.9% perplexity overhead over the public Mamba checkpoints on WikiText-103. Ablations attribute the gain chiefly to spectral normalisation of the selective projections and to step clamping.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Selective state-space models, Hidden-state poisoning, LipMamba, Lipschitz constant, Spectral normalisation, State-retention bound, PAC-Bayesian bound, Adversarial robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EkGxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LipMamba combines constraints on selective state-space dynamics with robustness-oriented fine-tuning to mitigate hidden-state poisoning. The paper provides theoretical analyses and reports substantial improvements on RoBench-25 and HarmBench with a modest perplexity cost.

Critical strengths
  1. The paper addresses a concrete vulnerability in selective SSMs and proposes a defense that directly modifies the underlying state dynamics.
  2. The reported robustness gains are substantial at both 130M and 370M scales. Component ablations, step-size sensitivity analysis, and measurements of clean performance and latency provide useful initial evidence about the method’s benefits and costs.
Critical weaknesses
  1. A matched fine-tuned unconstrained baseline is missing. Section 6 evaluates unconstrained Mamba as released, while LipMamba receives additional WikiText-103 adaptation. The “without adversarial training” ablation still evaluates an adapted constrained model, rather than an unconstrained model receiving equivalent fine-tuning. Consequently, attributing the improvement from 23.4% to 61.9% to “the architecture alone” is not sufficiently supported. The comparison provides evidence for the overall pipeline, but does not isolate the constraints’ contribution. An unconstrained Mamba baseline under matched fine-tuning conditions is needed.

  2. The evaluation is unclear about attacks adapted to the defense. Table 1 and the limitations section refer to published HiSPA triggers, but do not clearly establish whether they were reoptimized against each trained LipMamba model. Since the defense modifies the mechanism targeted by the original attacks, resistance to existing triggers may not imply resistance to adapted attacks. The authors should specify the target checkpoints, attack objectives, budgets, and restarts, including whether attacks optimize downstream answer errors. Embedding-space PGD addresses a different threat model and does not resolve this uncertainty.

  3. The experimental coverage does not adequately characterize the robustness–utility trade-off. The discrete-attack evidence centers on HiSPA, while clean utility is assessed mainly through RoBench-25 accuracy and WikiText-103 perplexity. These measurements are useful, but do not directly test whether constraining state updates impairs useful forgetting, revision of previously supplied information, or context switching. Such effects are plausible because input-dependent information selection is central to Mamba’s functionality; they are not established failures of LipMamba. A focused evaluation of normal information-update behavior would help determine whether the reported perplexity cost adequately reflects the practical trade-off.

  4. The presentation makes the central contribution difficult to assess. State-retention analysis, continuous-input sensitivity, GloRo-style training, and PAC-Bayesian regularization are introduced without a sufficiently clear account of how each supports the primary discrete-poisoning objective. Figure 2 further juxtaposes local-radius-based LL-Acc with direct PGD accuracy for SpectralGuard. Although disclosed in the caption, these metrics measure different properties. A concise overview connecting design choices, theoretical scope, and evaluation metrics would substantially improve clarity.

Minor comments.

  • The proof of Theorem 3 on page 14 is densely typeset. Separate displayed equations and explanations of the main steps would improve readability.
  • Page 2 promises scripts to regenerate all tables and figures, whereas docs/REPRODUCIBILITY.md in anomyous github states that Table 1’s numbers were not produced by this repository at the time of writing. Please clarify how the released code relates to the reported experiments. This issue has not influenced my rating.
Questions for authors
  1. Matched training controls: How does unconstrained Mamba perform under the same fine-tuning conditions? What improvement remains attributable to the proposed constraints?
  2. Adaptive attack evaluation: Were HiSPA triggers reoptimized against each trained LipMamba checkpoint? Please specify the objectives, budgets, and restarts, including whether optimization targets downstream answer errors.
  3. Robustness–utility trade-off: How do the constraints affect normal information revision and context switching? Can the authors provide a focused evaluation beyond perplexity and clean retrieval accuracy?
4 comments
AuthorsOpenReview ↗

Matched fine-tuning control, attack protocol, fact-revision test and corrected reproducibility statement

Thank you for the careful review and for checking the repository. Your questions shaped the new experiments directly.

W1 / Q1. Matched training controls.

We ran the control you asked for. An unconstrained Mamba-130M and the constrained model are fine-tuned from the same public checkpoint with the same data, optimiser, steps and seeds, with and without adversarial training (general response, Table 2 of the revision). Under adapted triggers of length 8, accuracy is 20.8% (unconstrained) and 5.6% (constrained) with clean fine-tuning, and 71.5% and 56.3% with adversarial fine-tuning. The improvement is therefore attributable to adversarial training, and we withdraw the sentence about the 61.9% row. What the constraints contribute is the state-level guarantee, which we confirm by measurement when training from scratch (retention 0.64 against 0.03). One detail: at 130M the constrained arms use constants calibrated to the checkpoint, a procedure the revision introduces so that the constraints can be added to a pretrained model without changing its function.

W2 / Q2. Were triggers re-optimised?

In the revision, yes, always. Target: each fine-tuned checkpoint and each example separately. Vocabulary: all tokens except names, answer words and special tokens. Attacks: eight random triggers; a zero-shot trigger repeating the largest-step token among 1024 candidates; greedy coordinate gradient search on the answer margin (40 iterations, 32 exactly evaluated candidates each) and on the state norm. Evaluation: 48 held-out examples per seed and trigger length in {8, 16, 32}; an example is robust only if all four attacks fail. The earlier comparison under published triggers has been removed.

W3 / Q3. Useful forgetting, revision and context switching.

Both studies test this directly, and the answer is encouraging. Every context contains a sentence that re-assigns one fact, and in "update" examples the question is about that fact, so the model must return the revised value. Fact-update accuracy is 100% for the constrained and the unconstrained model alike at 130M, with the same WikiText perplexity (17.7 against 17.6), and 100% in the small study. Bounding the step does not impair revision of stored information in these tasks. A broader revision benchmark in natural language is part of the evaluation we are extending for the camera-ready version.

W4. Presentation.

The method section now opens with an overview table: for each component, the quantity it controls, the formal result, the threat surface (token insertion or continuous perturbation), and whether it is proved or measured. The experiments are reorganised around the two matched studies, and the figure that mixed a detector's PGD accuracy with our radius has been removed.

Minor: proof of Theorem 3. Rewritten as four displayed steps: error recurrence, term-by-term bounds, unrolling, readout.

Minor: reproducibility.

You are right, and we apologise: the sentence on page 2 promised more than the repository delivered. We corrected this in two ways. First, the results that the repository could not regenerate (Tables 1 and 2 of the submission and the figures from the same runs) are no longer in the paper. Second, the revised statement lists exactly what the released scripts regenerate: every derived constant, the analytical depth curve, and both matched studies with per-seed outputs (the small one on a CPU in about two hours, the 130M one on a single 16 GB GPU).

In progress for the camera-ready version: the remaining seed and the full ablation at 130M, tightened constants that make the retention bound informative at scale, and adapted-attack evaluation on RoBench-25 and HarmBench.

Thanks for the additional experiments and clarifications. The matched baselines, adaptive attacks, and fact-update test address most of my original concerns. However, the new results leave me less convinced about the benefit of the constraints.

The reported mean results at 130M are:

Model and trainingPACC@8 ↑ASR@8 ↓
Unconstrained, clean fine-tuning20.8%79.2%
Constrained (calibrated), clean fine-tuning5.6%94.4%
Unconstrained, adversarial fine-tuning71.5%28.5%
Constrained (calibrated), adversarial fine-tuning56.3%43.7%

Adversarial training helps both models, but the constrained model has lower mean accuracy in both comparisons. These results do not show an additional behavioral robustness benefit from the constraints.

I understand the revised emphasis on state-level guarantees, and I think these are worth studying. However, it remains unclear how preserving the state helps protect the final predictions. With the earlier attribution of robustness gains to the architecture withdrawn, this connection becomes important to assessing the remaining contribution.

For now, I will keep my score.

AuthorsOpenReview ↗

what the constraints do and do not protect

Thank you for reading the new results closely and for the clear follow-up. We agree with your reading of the table, and we would rather answer the question you raise directly than argue with the numbers.

On the table. You are right: under matched training the constrained model does not have higher accuracy under adapted triggers, with or without adversarial fine-tuning. The revision does not claim a behavioural robustness benefit from the constraints, and we withdrew the earlier attribution for exactly this reason. (The constrained adversarial arm has two seeds so far; the third is being added, and we do not expect it to change this conclusion.)

How does preserving the state relate to protecting the prediction? The prediction is read from the state through a projection that itself depends on the current input. A trigger can therefore change the answer through three channels:

  1. Erasure: shrinking the state so that earlier context is lost. This is the mechanism that defines hidden-state poisoning.
  2. Injection: adding new content to the state.
  3. Readout steering: changing how the state is read, without changing much of what it holds.

The retention bound closes channel 1, and only channel 1. That is a necessary condition for protecting the prediction, not a sufficient one, and our matched results show the difference in practice: at 130M a search that minimises the state norm changes no answer of any fine-tuned model, while a search on the answer margin, which works through channels 2 and 3, changes most of them. We consider this separation one of the useful outcomes of the experiments you asked for: it shows that defences and evaluations aimed at state collapse alone address the wrong channel once a model is fine-tuned.

What connects the two, and what is missing. The paper already bounds the other quantities that enter the prediction: the component of the state along its pre-trigger direction (Corollary 5), the norm of what a trigger can inject per token, and the gain of the readout projection. Together these give a condition of the form "if the answer margin exceeds a bound built from these three quantities, a trigger of that length cannot change the answer". With the constants a pretrained model admits, that bound is not yet informative, and we say so. Making it informative is the concrete next step, and it is the purpose of the experiment we are running now: fine-tuning while the step and eigenvalue bounds are tightened gradually to the regime of the stated constants, to measure whether a non-trivial bound can coexist with the pretrained model's utility.

What we think remains as the contribution. A closed-form analysis of the selective scan with the exact condition for a bounded state; a guarantee, confirmed by measurement when training from scratch, that removes the erasure channel; a calibration procedure that adds the constraints to a pretrained model at no cost in accuracy or perplexity; and a training-matched protocol with adapted attacks that shows which channel is actually binding. We agree that this is a narrower claim than the submission made, and we appreciate that your questions are what made it precise.

We understand your decision to keep the score, and we thank you again for the time you have put into this paper.

Thanks for the clarification. I appreciate the state-level contribution. However, the bound connecting these guarantees to robust predictions is not yet informative with the pretrained model’s constants, and the matched experiments show no additional robustness benefit from the constraints.

I will keep my current score for now, but would be happy to reconsider when the constraint-tightening results are available.

Reviewer_MwDKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LipMamba with spectral normalization, bounded eigenvalues, and a bounded discretization step. It derives Lipschitz, state-retention, and PAC-Bayes results and evaluates HiSPA robustness. The engineering idea is interesting, but the main certification is vacuous and the empirical gains are confounded by adversarial training. I do not recommend acceptance in the current form.

Critical strengths
  • It identifies a concrete state-erasure mechanism in selective SSMs.
  • The parameterization enforces several stability constraints by construction.
  • The paper clearly distinguishes global bounds, local estimates, and state-norm guarantees.
  • The ablations show that spectral normalization and step clamping matter.
Critical weaknesses
  • The global certificate is vacuous. With the default constants, the paper reports a per-block Lipschitz constant of about 1.1×10^8 and a 24-block product around 10^193, yielding a GloRo radius below 10^-10.

  • The PAC-Bayes bound is not instantiated. The theorem requires a valid upper Lipschitz bound, but the reported L_loc is explicitly only a lower estimate. The training objective is therefore a heuristic rather than a numerical PAC-Bayes guarantee.

  • State retention is not behavioral robustness. At the default constants, the theorem certifies only about 1 token at 50% retention and up to 6 tokens against 95% erasure, while attacks use length 24. It also does not control the readout C_t^T h_t.

  • The empirical comparison is confounded. LipMamba-370M reaches 85.3% poisoned accuracy versus 23.4% for Mamba, but the paper reports 61.9% for the constrained architecture without the adversarial objective. Baselines are not retrained with the same objective and data.

  • The attack evaluation is narrow. Experiments use published HiSPA triggers and PGD-40, without adaptive attacks against the full constrained recurrence or attacks targeting the readout.

  • Local robust accuracy is not certified. The reported 80.4% [email protected] uses an empirical local estimate and should not be presented alongside certified robustness without stronger qualification.

Questions for authors

See Weakness.

1 comment
AuthorsOpenReview ↗

Point-by-point response: scope of the guarantees, matched controls and adaptive attacks

Thank you for the precise review. We ran the experiments your points call for, and the revision reports them in full. We answer each weakness in turn and say what is being completed for the camera-ready version.

W1. The global certificate is vacuous.

We agree on the number: 1.1×1081.1\times10^{8} per block and 1019310^{193} over 24 blocks, from which no radius follows. The revision presents Theorem 3 for what it does deliver. It is the first closed-form constant for an input-dependent scan, and it identifies Δmin⁡>0\Delta_{\min}>0 as the exact condition that bounds the state, which explains the error explosion reported for selective SSMs. The overview table marks it "proved; vacuous at depth".

W2. The PAC-Bayes bound is not instantiated.

Agreed. It is now a proposition that motivates the form of the regulariser, and the objective is described as a heuristic. A depth-independent local bound that would allow an instantiation is stated as an open problem.

W3. State retention is not behavioural robustness.

We now measure both, at two scales. Trained from scratch with the stated constants, the constrained state keeps at least 0.64 of its norm under an adapted trigger where the unconstrained state falls to 0.03, so the theorem holds in practice. The same study shows that retention and behaviour are distinct: the readout can be steered without erasing the state. At 130M the point is sharper and, we think, useful to the field: a token search on the state norm (the hidden-state-poisoning objective) changes no answer of any fine-tuned model, while a search on the answer margin does. The revision states this and names the extension of the bound to the readout as the main theoretical next step.

W4. The empirical comparison is confounded.

We removed that comparison and replaced it with a matched one. An unconstrained and a constrained Mamba-130M are fine-tuned from the same public checkpoint with the same data, optimiser, steps and seeds. Utility is identical (100% accuracy, perplexity 17.6 to 17.7). Under adapted triggers of length 8, accuracy is 20.8% (unconstrained) and 5.6% (constrained) with clean fine-tuning, and 71.5% and 56.3% with adversarial fine-tuning. So the robustness comes from adversarial training, and we withdraw the attribution to the architecture.

W5. The attack evaluation is narrow.

Every attack in the revision is adapted to each checkpoint and each example, end to end through the clamp and the readout: greedy coordinate gradient search with 40 iterations of 32 exactly evaluated candidates over about 50,000 admissible tokens, on the answer margin and on the state norm, plus random and zero-shot triggers. An example counts as robust only if all four fail.

W6. Local robust accuracy is not certified.

Agreed. It is no longer reported as a result; the paper keeps only the definition, with the statement that such radii are empirical.

What remains as the contribution, and what is in progress. (i) The closed-form analysis of the selective scan and its boundedness condition; (ii) a parameterisation that provably and measurably keeps the state norm bounded, with a calibration procedure that adds it to a pretrained model at no utility cost; (iii) a training-matched protocol with adapted attacks that shows which part of robustness the constraints supply and which part needs adversarial training. For the camera-ready version we are tightening the calibrated constants during fine-tuning towards the regime where the retention bound is informative at 130M, completing the ablation over three seeds, and extending the adapted-attack evaluation to RoBench-25 and HarmBench.

Reviewer_d3ccOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors proposed a Lipschitz-constrained mamba model as a way to defend against attacks, evaluated using the HarmBench protocol, showing a large decrease in Attack Success Rate while keeping performance high enough.

Critical strengths

The reproducibility of the paper is ensured as the code and the data is publicaly available, the experiments are clear, and the results are supporting the claims.

Critical weaknesses

None

Questions for authors

Could you include an ablation table similar to Table 2 but on HarmBench, including the ASR metric? This ablation support the best parameters for performance optimization, but does not inform about the attack robustness.

1 comment
AuthorsOpenReview ↗

Ablation with attack success rate under re-optimised attacks, and the HarmBench ablation in progress

Thank you for the supportive review and for the question, which led to a better ablation than the one we submitted.

Q: an ablation like Table 2, but on HarmBench with ASR.

The revision reports attack success rate (ASR) under attacks that are re-optimised against every model, in two matched studies. ASR is the fraction of clean-correct inputs whose answer the strongest attack changes, at trigger length 8.

130M, fine-tuned from the public checkpointPACC@8ASR@8
Unconstrained, clean fine-tuning20.879.2
All constraints (calibrated), clean fine-tuning5.694.4
Unconstrained, adversarial fine-tuning71.528.5
All constraints (calibrated), adversarial fine-tuning56.343.7
Small model, trained from scratchPACC@8ASR@8
All constraints30.769.3
without spectral normalisation45.354.7
without step clamp24.275.8
without eigenvalue reparameterisation3.595.7
All constraints, adversarial training98.21.8

What this tells us about the safety metric:

  • Adversarial training is the component that lowers ASR, by far more than any single constraint (to 1.8% from scratch, and from 79.2% to 28.5% at 130M).
  • The eigenvalue reparameterisation and the step clamp matter for the constrained model: removing them raises ASR to 95.7% and 75.8%.
  • Spectral normalisation trades differently than we reported: removing it lowers ASR in the small study. We report this ordering as measured.

The HarmBench version of this ablation is in progress. We did not want to approximate it in the response period: the 130M single-constraint ablation currently has one seed (PACC@8 of 18.8% without spectral normalisation and 8.3% without the clamp, against a seed-to-seed range of 0 to 12.5% for the full model), so we draw no conclusion from it yet. For the camera-ready version we are completing the three-seed ablation at 130M and the adapted-attack evaluation on HarmBench with ASR, so that the ablation speaks to the safety benchmark directly, as you asked.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New training-matched experiments at 130M parameters and a revised empirical section

We thank the three reviewers for careful and constructive reviews. They converge on one request: show what the constraints contribute under matched training and under attacks adapted to the defended model. We ran exactly these experiments, and the revised paper is rebuilt around them. Every number in it is regenerated by a released script, with per-seed outputs.

1. Matched fine-tuning at 130M parameters (new Section 6.1, Table 2, Appendix J). Every arm starts from the public Mamba-130M checkpoint and receives the same data, optimiser, 400 steps and seeds. Task: natural-language fact recall with a later re-assignment of one fact. Attacks are re-optimised per example against each fine-tuned checkpoint, over the whole vocabulary except names, answer words and special tokens: random triggers, a zero-shot largest-step trigger, and greedy coordinate gradient search on the answer margin (end to end) and on the state norm.

ModelSeedsCleanUpdatePPLPACC@8 / 16 / 32
Public checkpoint, not fine-tuned325.019.026.40.0 / 0.0 / 0.0
Unconstrained, clean fine-tuning3100.0100.017.620.8 / 0.7 / 0.0
Constrained (calibrated), clean3100.0100.017.75.6 / 0.0 / 0.0
Unconstrained, adversarial3100.0100.017.671.5 / 15.3 / 0.7
Constrained (calibrated), adversarial299.8100.017.656.3 / 5.2 / 1.0

Values in %, mean over seeds, 48 test examples per seed and trigger length. PACC@ℓ\ell is accuracy under the worst of the four attacks at trigger length ℓ\ell.

2. Controlled study from scratch (Section 6.2, Table 3, Appendix K). A two-block model trained with the constants stated in the paper. Under an adapted trigger the constrained state keeps at least 0.64 of its norm where the unconstrained state falls to 0.03, as Theorem 4 predicts. PACC@8 is 98.2% (constrained) and 100.0% (unconstrained) with adversarial training.

3. Scope of the empirical section. We removed the comparison with released checkpoints under published triggers (Tables 1 and 2 of the submission and the figures drawn from the same runs). As the reviewers observed, it was not training-matched, and we could not release its run logs within the response period. The matched studies replace it.

What the evidence establishes

  • The theory holds where its constants hold. Trained from scratch, the clamp keeps the state bounded away from zero exactly as proved, which removes the collapse mechanism of hidden-state poisoning.
  • The constraints can be added to a pretrained Mamba at no utility cost. The constants stated in the paper suit training from scratch; projected onto a pretrained checkpoint they change its function (Table 2, last rows). The revision therefore introduces constants calibrated to the checkpoint, with which accuracy, fact-update accuracy and perplexity match the unconstrained model.
  • Adversarial fine-tuning supplies behavioural robustness for short triggers (20.8% to 71.5% at ℓ=8\ell=8).
  • The binding attack targets the answer margin, not the state norm. A search on the state norm changes no answer of any fine-tuned model. This matters for anyone evaluating defences against hidden-state poisoning.
  • Stated plainly: the constraints alone do not raise accuracy under adapted triggers, and we withdraw the sentence attributing part of the gain to "the architecture alone". At 130M the calibrated constants are still too wide for the retention bound to be informative.

Experiments in progress for the camera-ready version

  • the third seed of the constrained adversarial arm and the complete single-constraint ablation at 130M;
  • tightening the calibrated constants during fine-tuning towards the regime where the retention bound is informative, and longer fine-tuning with the stated constants;
  • adapted-attack evaluation on RoBench-25 and HarmBench, including the ASR ablation;
  • the same protocol at 370M parameters.

Other changes. An overview table links each component to its result and its status (proved or measured); the PAC-Bayes statement is now a proposition; the proof of Theorem 3 is displayed step by step; the empirical local-Lipschitz accuracy is no longer reported as a result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.