- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces CountTrigger which is a backdoor mechanism in which LLM behavior changes when the number of qualifying items reach a specified threshold. The authors evaluate whether models can learn these triggers, whether they learn the intended counting rule and whether internal counting representations actually affect the final behaviour. Experiments cover three datasets, several LLMs and additional banking and tool use scenarios. The authors also use auxiliary finite state supervision and activation interventions to study how counting information is represented.
Critical strengths
- The paper shows that models can achieve almost perfect attack success rates on standard test cases whilst learning incorrect counting rules. This is a useful finding as it demonstrates that high ASRs alone do not establish that the intended trigger condition has been learned. The additional counterfactual experiments and held-out renderer results are useful. They show that carefully constructed supervision can improve the reliability and generalization of this behaviour.
- The authors examine internal counting representations using auxialiary supervision and causal interventions instead of relying on output behavior. The finding that readable count representations do not necessarily control the final action is interesting. The use of matched controls and different intervention locations adds value to the experimental analysis.
Critical weaknesses
- The main contribution is introducing the number of qualifying items as another backdoor trigger. However, this has already been explored in non-lexical triggers based on structural and numerical properties such as MetaBackdoor [1] (studies sequence length conditions) and Turn-based structural triggers [2] (dialogue position). CountTrigger seems to differ by counting selected items rather than total tokens or turns. But this seems like a relatively small extension. The authors do not establish why this distinction leads to fundamentally new vulnerabilities or stronger attacks. The theoretical contribution is also limited. The authors explicity build on existing Transformer expressivity results and assume that qualifying units can already be identified using first order logic. Showing that a fixed number of such units can be counted does not appear to provide a new theoretical insight.
- Most experiments use highly controlled inputs with explicitly defined qualifying and non-qualifying events. The new rebuttal experiments substantially improve generalization results. They show nearly 100% activation and 0% below-threshold FPR on unseen renderers. But, these improvements come from a very different training setup. There are many more counterfactual examples (38,500 vs 7,000) and explicit control over event position and payload length. Whilst these demonstrate models can learn the intended threshold condition under carefully constructed supervsion, they do not establish that the learned rule transfers. The held-out event-composition experiments are also limited to familiar event types and fixed structures. Further, the original tool-use experiments show name-only and duplicate tool objects can activate the backdoor. These findings raise questions on whether the model learns a meaningful notion of valid qualifying objects or reamin dependent on specific input representations.
- The mechanism analysis is also not convincing. The strong state-decoding results comes from models that are trained explicitly with auxiliary finite-state supervision. This does not establish that normally backdoored models would learn the same counting process. Also, changing the decoded count state at event positions produces almost no change in final action. Even the late-position interventions are not consistently count-specific. Overall, I feel the experiments are better at demonstrating limitations of the trigger rather than establishing that it learns reliable and transferable counting behaviour in realistic systems.
Questions for authors
Answered in rebuttal.
6 comments
Thanks for the detailed response and addiitonal results. I appreciate the clarification regarding the distinction between selective cardinality and existing structural triggers. I also acknowledge now that one of the comparisons requested was already in the appendix, which I had missed. The new results are encouraging and address an important part of my generalization concerns. However, I'd appreciate some clarification on two more points.
- The new held-out-renderer experiments achieve almost 100% activation compared with 34% and 10% in the original manuscript. Could you clarify the differences in training config between these and report the corresponding below-threshold FPR? High positive activation alone does not establish that the counting predicate has been learned.
- The new causal intervention results demonstrate that steering a count-correlated representation changes the logit margin. However, since count-four and five examples have different Action targets, how do authors distinguish a representation of accumulated count from a representation of the learned decision? Do these interventions also produce changes in actual generated actions?
I agree that selective cardinality is a distinct conditioning variable and that the new results definitely strenghten the emprical contribution. My main remaining conerns is where this evidence establishes a generalizable counting mechanism rather than a trigger decision learned from the specific training setup.
Boundary of our generalizable counting mechanism
We thank the reviewer for this thoughtful follow-up. We appreciate the question on whether our evidence supports a generalizable counting mechanism, which helps us clarify the scope of our study.
1. Different training configuration
Thank you for asking us to clarify this comparison. The original 34%/10% results and the new held-out-renderer results come from different training configurations, with the most important difference being the training-data construction.
The key differences are summarized below:
| Original O-enriched FSM | New O-rich with no FSM | |
|---|---|---|
| Training objective | Output SFT + explicit FSM state supervision at event boundaries | Output-only SFT; no state head (FSM) |
| Event-level counterexamples | 7,000 events | 38,500 events |
| events control | No jointly fixed control of event position and payload length | Explicit control of cardinality, event position, and payload length |
| Count-level below-threshold examples | 7,000 examples with | 7,700 examples with |
| Final count-state decoding | 100% | N/A (no state head) |
| Held-out-renderer activation | 34%/ 10% | 99.5/ 100% |
Although below-threshold inputs are comparable (7,000 vs. 7,700), the new construction provides substantially more event-level counterexamples (38,500 vs. 7,000 O events).
The old FSM (|U|U|O|)supervision provides labels for intermediate accumulation states at designated event-end positions. It can make these states highly readable, as reflected by the 100% final count-state decoding accuracy. However, identifying an event boundary does not establish whether the complete event satisfies the qualifying predicate . The model must still inspect the event content, distinguish from , and accumulate only qualifying events. A boundary-local state label alone does not guarantee that these event-level decisions are learned or reliably used by the downstream Action policy.
The new construction fixes seven event slots and controls U/O composition, count, position, and payload length. This provides richer counterfactual supervision and makes shortcuts based on slot count and surface statistics harder to exploit.
The new models achieve 99.5–100% activation at , with 0% below-threshold FPR:
| Unseen Renderer(new model) | Seed 30: c5 ASR / below FPR | Seed 32: c5 ASR / below FPR |
|---|---|---|
| Arrow | 100% / 0% | 100% / 0% |
| Checkbox | 100% / 0% | 99.5% / 0% |
Below-threshold FPR is computed over all examples, with 400 examples per renderer and seed.
Importantly, the same unseen renderer appears in both positive and negative examples( and ). Thus, the 0% FPR rules out a simple renderer-presence shortcut: the renderer alone does not trigger the Action below the threshold, while the model activates almost perfectly when .
Together with the controlled construction, these results support a more robust threshold-conditioned Action policy, rather than unconditional activation associated with a particular renderer.
2. Generalizable counting mechanism
Our experiments demonstrate two forms of generalization within the scope of qualifying-event distributions represented during training:
| Generalization | Training (seen) | Testing (unseen) |
|---|---|---|
| Renderer generalization | - [U1: event] | -> [U1: event] |
| Event-composition generalization | <br/>AAABB(3A+2B)/ AABBB(2A+3B) → Action_M | AAAAB (4A+1B) / ABBBB (1A+4B): → Action_M |
-
Renderer / structural generalization: The event content and qualifying structure remain familiar, while only the renderer changes. O-rich supervision supports transfer across these structurally similar renderings.
-
Event-composition generalization: We demonstrate that the model generalizes from AAABB/AABBB to held-out AAAAB/ABBBB; both achieve 100% accuracy in both seeds, while mixed (c=4) controls have 0% false rate.
Because and are associated with different Action targets, these interventions do not isolate accumulated count from the learned decision. Single-site interventions at the L24 prompt-final position modulate the target logit margin , but produce 0% final Action flips. We therefore interpret the localized direction as count-correlated rather than as a causally sufficient, transferable counting direction.
Generalization boundary: We do not assume that a counting rule learned for one event type automatically transfers to other types. This is consistent with Hasani et al. (CVPR 2026), who report type-specific resetting counters in polytypic lists rather than a single global counter. These findings suggest that cross-type aggregation should not be taken for granted.
We sincerely thank the reviewer again for the constructive feedback, which have helped us substantially strengthen this work.
Clarifying CountTrigger’s Novelty and Contribution: From a New Trigger Paradigm to the Scientific Problem of Cardinality and Selective Accumulation in LLMs
W1/Q1: Clarifying CountTrigger’s novelty and scientific contribution
We respectfully disagree with the characterization of CountTrigger as a relatively small extension that merely “counts selected items rather than total tokens or turns.” The similarity is superficial: the trigger families require different conditioning variables and computations.
The key distinction is therefore the learning capability required. Length and turn are global structural variables determined by the input, whereas CountTrigger requires the model to construct a latent conditioning variable by identifying qualifying events and selectively accumulating them.
The matched comparison was already reported in Appendix A.1, Table 2 of our original manuscript.
From an attack perspective, CountTrigger remains effective as long as the cardinality predicate is preserved, exposing a distinct defense blind spot.
However, CountTrigger should not be evaluated solely from an attack perspective. As shown in Fig. 1, CountTrigger is dual-use, meaning that in some settings we may want to deliberately use rather than eliminate it. More fundamentally, CountTrigger exposes a deeper model-capability question: whether LLMs can acquire and use selective cardinality from sparse behavioral supervision.
Why CountTrigger raises a harder scientific problem—formal analysis is necessary
Prior work shows that counting, OOD counting, and systematic rule generalization are already non-trivial for Transformers/LLMs (Bhattamishra et al., 2020; Delétang et al., 2023; Chang & Bisk, 2024; Yehudai et al., 2024). CountTrigger is harder still: the model receives no count labels, no U/O labels, and no instruction to count, yet must infer what counts, selectively accumulate it, and use the resulting cardinality for behavior from downstream supervision alone.
This is also why Sec. 3.2 is necessary. Not every counting rule is automatically representable or learnable. Even simple token counting exhibits capacity-dependent limits (Yehudai et al., 2024), and Transformer learning/generalization varies substantially across formal languages (Bhattamishra et al., 2020). Consider a trigger that activates when CU(x) is prime: fitting observed counts 2,3,5,7 does not imply generalization to 11,13, and such an unbounded predicate is not covered by our fixed-k first-order construction.
Our fixed-k construction is representable under the stated unit-definability assumptions. Without such grounding, empirical failure would be uninterpretable: it could reflect either insufficient supervision or an unattainable target.
W2/Q2: Why fall to 34% and 10%?
LLMs are known to be sensitive even to meaning-preserving formatting changes (Sclar et al., 2024). The 34%/10% values came from one Seed-32 checkpoint on unseen
arrow/checkboxrenderers.We tested whether this sensitivity can be overcome by retraining no-FSM models with both renderers excluded from SFT:
arrowcheckboxThus, cross-renderer generalization is coverage-dependent rather than an inherent limitation of CountTrigger.
W3/Q3: Causal connection without FSM supervision
Yes. FSM supervision was introduced as an analytic scaffold, not a prerequisite for CountTrigger. An apparently correct counting response does not necessarily imply acquisition of the underlying cardinal principle (Sarnecka & Carey, 2008).
We directly test independently trained no-FSM Count-SFT models, which receive the same behavioral Action supervision but no auxiliary finite-state labels. d=E[h∣c=5]−E[h∣c=4]. At the prompt-final representation, we add +d to c=4 examples or −d to c=5 examples, and measure the change in the target-vs.-non-target Action logit margin.
Thus, even without finite-state supervision, moving the internal representation toward the trigger side increases target-Action preference, while moving it toward the non-trigger side decreases it. This establishes a causal contribution of a count-aligned representation to downstream behavior.