acceptodds
Under review as a conference paper at ICLR 2027

CountTrigger: What Do LLMs Actually Learn to Count in Cardinality-Conditioned Backdoors?

Abstract

Large language models increasingly operate over structured collections of tools, records, evidence items, and workflow events. We ask whether the cardinality of task-defined structures can become a latent condition for model behavior. We introduce CountTrigger, a family of structural cardinality predicates specified by what to count, where and how counting is applied, and when the resulting count activates behavior. We first give a formal grounding for fixed threshold, exact-count, and band predicates under explicit unit-definability assumptions, separating representability from learnability. Empirically, CountTrigger activation is broadly learnable across 20 model–task settings, while counterfactual controls show that high attack success can coexist with reliance on surrogate structural cues. Matched finite-state supervision makes accumulation states highly decodable, yet multi-seed activation interventions reveal a representation–behavior gap: event-level states rarely flip the final action, whereas late decision-position interventions produce substantially larger effects with recipient-dependent asymmetry. Finally, the same primitive transfers from a controlled setting to native tool registries and banking-event histories, including sharp count boundaries in xLAM and source-disjoint τ-Banking workflows. Together, the results connect formal expressivity, empirical acquisition, internal mechanism, and downstream behavioral control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Security, Backdoor Attacks, Structural Backdoors, Counting, Structural Cardinality, Mechanistic Interpretability, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fohjOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces CountTrigger which is a backdoor mechanism in which LLM behavior changes when the number of qualifying items reach a specified threshold. The authors evaluate whether models can learn these triggers, whether they learn the intended counting rule and whether internal counting representations actually affect the final behaviour. Experiments cover three datasets, several LLMs and additional banking and tool use scenarios. The authors also use auxiliary finite state supervision and activation interventions to study how counting information is represented.

Critical strengths
  1. The paper shows that models can achieve almost perfect attack success rates on standard test cases whilst learning incorrect counting rules. This is a useful finding as it demonstrates that high ASRs alone do not establish that the intended trigger condition has been learned. The additional counterfactual experiments and held-out renderer results are useful. They show that carefully constructed supervision can improve the reliability and generalization of this behaviour.
  2. The authors examine internal counting representations using auxialiary supervision and causal interventions instead of relying on output behavior. The finding that readable count representations do not necessarily control the final action is interesting. The use of matched controls and different intervention locations adds value to the experimental analysis.
Critical weaknesses
  1. The main contribution is introducing the number of qualifying items as another backdoor trigger. However, this has already been explored in non-lexical triggers based on structural and numerical properties such as MetaBackdoor [1] (studies sequence length conditions) and Turn-based structural triggers [2] (dialogue position). CountTrigger seems to differ by counting selected items rather than total tokens or turns. But this seems like a relatively small extension. The authors do not establish why this distinction leads to fundamentally new vulnerabilities or stronger attacks. The theoretical contribution is also limited. The authors explicity build on existing Transformer expressivity results and assume that qualifying units can already be identified using first order logic. Showing that a fixed number of such units can be counted does not appear to provide a new theoretical insight.
  2. Most experiments use highly controlled inputs with explicitly defined qualifying and non-qualifying events. The new rebuttal experiments substantially improve generalization results. They show nearly 100% activation and 0% below-threshold FPR on unseen renderers. But, these improvements come from a very different training setup. There are many more counterfactual examples (38,500 vs 7,000) and explicit control over event position and payload length. Whilst these demonstrate models can learn the intended threshold condition under carefully constructed supervsion, they do not establish that the learned rule transfers. The held-out event-composition experiments are also limited to familiar event types and fixed structures. Further, the original tool-use experiments show name-only and duplicate tool objects can activate the backdoor. These findings raise questions on whether the model learns a meaningful notion of valid qualifying objects or reamin dependent on specific input representations.
  3. The mechanism analysis is also not convincing. The strong state-decoding results comes from models that are trained explicitly with auxiliary finite-state supervision. This does not establish that normally backdoored models would learn the same counting process. Also, changing the decoded count state at event positions produces almost no change in final action. Even the late-position interventions are not consistently count-specific. Overall, I feel the experiments are better at demonstrating limitations of the trigger rather than establishing that it learns reliable and transferable counting behaviour in realistic systems.
Questions for authors

Answered in rebuttal.

6 comments
AuthorsOpenReview ↗

Clarifying CountTrigger’s Novelty and Contribution: From a New Trigger Paradigm to the Scientific Problem of Cardinality and Selective Accumulation in LLMs

W1/Q1: Clarifying CountTrigger’s novelty and scientific contribution

We respectfully disagree with the characterization of CountTrigger as a relatively small extension that merely “counts selected items rather than total tokens or turns.” The similarity is superficial: the trigger families require different conditioning variables and computations.

Trigger familyRepresentativeConditioning signal
MetaBackdoor [1]L(x)≥kL(x)\ge kInput length
TST [2]turn⁡(x)=k\operatorname{turn}(x)=kDialogue turn
CountTriggerCU(x)=∑i1[U(ei)]≥kC_U(x)=\sum_i\mathbf{1}[U(e_i)]\ge kUOU→CU=2UOU \to C_U = 2

The key distinction is therefore the learning capability required. Length and turn are global structural variables determined by the input, whereas CountTrigger requires the model to construct a latent conditioning variable by identifying qualifying events and selectively accumulating them.

The matched comparison was already reported in Appendix A.1, Table 2 of our original manuscript.

Trigger (ASR)Undefended %ONION-style %Truncation %Clean-SFT %
Fixed-token100.000.00100.0025.50
sentence-level97.7543.7574.130.00
Length-threshold99.88100.007.630.00
CountTrigger100.0099.1399.88100.00

From an attack perspective, CountTrigger remains effective as long as the cardinality predicate is preserved, exposing a distinct defense blind spot.

However, CountTrigger should not be evaluated solely from an attack perspective. As shown in Fig. 1, CountTrigger is dual-use, meaning that in some settings we may want to deliberately use rather than eliminate it. More fundamentally, CountTrigger exposes a deeper model-capability question: whether LLMs can acquire and use selective cardinality from sparse behavioral supervision.

Why CountTrigger raises a harder scientific problem—formal analysis is necessary

Prior work shows that counting, OOD counting, and systematic rule generalization are already non-trivial for Transformers/LLMs (Bhattamishra et al., 2020; Delétang et al., 2023; Chang & Bisk, 2024; Yehudai et al., 2024). CountTrigger is harder still: the model receives no count labels, no U/OU/O labels, and no instruction to count, yet must infer what counts, selectively accumulate it, and use the resulting cardinality for behavior from downstream supervision alone.

This is also why Sec. 3.2 is necessary. Not every counting rule is automatically representable or learnable. Even simple token counting exhibits capacity-dependent limits (Yehudai et al., 2024), and Transformer learning/generalization varies substantially across formal languages (Bhattamishra et al., 2020). Consider a trigger that activates when CU(x)C_U(x) is prime: fitting observed counts 2,3,5,72,3,5,7 does not imply generalization to 11,1311,13, and such an unbounded predicate is not covered by our fixed-kk first-order construction.

Our fixed-kk construction is representable under the stated unit-definability assumptions. Without such grounding, empirical failure would be uninterpretable: it could reflect either insufficient supervision or an unattainable target.

W2/Q2: Why fall to 34% and 10%?

LLMs are known to be sensitive even to meaning-preserving formatting changes (Sclar et al., 2024). The 34%/10% values came from one Seed-32 checkpoint on unseen arrow/checkbox renderers.

We tested whether this sensitivity can be overcome by retraining no-FSM models with both renderers excluded from SFT:

Held-out rendererSeed 30Seed 32
arrow100%100%
checkbox100%99.88%

Thus, cross-renderer generalization is coverage-dependent rather than an inherent limitation of CountTrigger.

W3/Q3: Causal connection without FSM supervision

Yes. FSM supervision was introduced as an analytic scaffold, not a prerequisite for CountTrigger. An apparently correct counting response does not necessarily imply acquisition of the underlying cardinal principle (Sarnecka & Carey, 2008).

We directly test independently trained no-FSM Count-SFT models, which receive the same behavioral Action supervision but no auxiliary finite-state labels. d=E[h∣c=5]−E[h∣c=4].d=\mathbb{E}[h\mid c=5]-\mathbb{E}[h\mid c=4]. At the prompt-final representation, we add +d+d to c=4c=4 examples or −d-d to c=5c=5 examples, and measure the change in the target-vs.-non-target Action logit margin.

InterventionΔ\Delta target-Action margin
Toward trigger: c4→c5c4\rightarrow c5+0.038 to +0.270
Away from trigger: c5→c4c5\rightarrow c4−0.110 to −0.308
Random / orthogonal control≤0.032\le 0.032

Thus, even without finite-state supervision, moving the internal representation toward the trigger side increases target-Action preference, while moving it toward the non-trigger side decreases it. This establishes a causal contribution of a count-aligned representation to downstream behavior.

Thanks for the detailed response and addiitonal results. I appreciate the clarification regarding the distinction between selective cardinality and existing structural triggers. I also acknowledge now that one of the comparisons requested was already in the appendix, which I had missed. The new results are encouraging and address an important part of my generalization concerns. However, I'd appreciate some clarification on two more points.

  1. The new held-out-renderer experiments achieve almost 100% activation compared with 34% and 10% in the original manuscript. Could you clarify the differences in training config between these and report the corresponding below-threshold FPR? High positive activation alone does not establish that the counting predicate has been learned.
  2. The new causal intervention results demonstrate that steering a count-correlated representation changes the logit margin. However, since count-four and five examples have different Action targets, how do authors distinguish a representation of accumulated count from a representation of the learned decision? Do these interventions also produce changes in actual generated actions?

I agree that selective cardinality is a distinct conditioning variable and that the new results definitely strenghten the emprical contribution. My main remaining conerns is where this evidence establishes a generalizable counting mechanism rather than a trigger decision learned from the specific training setup.

AuthorsOpenReview ↗

Boundary of our generalizable counting mechanism

We thank the reviewer for this thoughtful follow-up. We appreciate the question on whether our evidence supports a generalizable counting mechanism, which helps us clarify the scope of our study.

1. Different training configuration

Thank you for asking us to clarify this comparison. The original 34%/10% results and the new held-out-renderer results come from different training configurations, with the most important difference being the training-data construction.

The key differences are summarized below:

Original O-enriched FSMNew O-rich with no FSM
Training objectiveOutput SFT + explicit FSM state supervision at event boundariesOutput-only SFT; no state head (FSM)
Event-level OO counterexamples7,000 OO events38,500 OO events
OO events controlNo jointly fixed control of event position and payload lengthExplicit control of cardinality, event position, and payload length
Count-level below-threshold examples7,000 examples with c<5c<57,700 examples with c<5c<5
Final count-state decoding100%N/A (no state head)
Held-out-renderer c=5c=5 activation34%/ 10%99.5/ 100%

Although below-threshold inputs are comparable (7,000 vs. 7,700), the new construction provides substantially more event-level counterexamples (38,500 vs. 7,000 O events).

The old FSM (|U|U|O|)supervision provides labels for intermediate accumulation states at designated event-end positions. It can make these states highly readable, as reflected by the 100% final count-state decoding accuracy. However, identifying an event boundary does not establish whether the complete event satisfies the qualifying predicate UU. The model must still inspect the event content, distinguish UU from OO, and accumulate only qualifying events. A boundary-local state label alone does not guarantee that these event-level decisions are learned or reliably used by the downstream Action policy.

The new construction fixes seven event slots and controls U/O composition, count, position, and payload length. This provides richer counterfactual supervision and makes shortcuts based on slot count and surface statistics harder to exploit.

The new models achieve 99.5–100% activation at c=5c=5, with 0% below-threshold FPR:

Unseen Renderer(new model)Seed 30: c5 ASR / below FPRSeed 32: c5 ASR / below FPR
Arrow100% / 0%100% / 0%
Checkbox100% / 0%99.5% / 0%

Below-threshold FPR is computed over all c=2,3,4c=2,3,4 examples, with 400 examples per renderer and seed.

Importantly, the same unseen renderer appears in both positive and negative examples(c=5c=5 and c<5c<5). Thus, the 0% FPR rules out a simple renderer-presence shortcut: the renderer alone does not trigger the Action below the threshold, while the model activates almost perfectly when c=5c=5.

Together with the controlled U/OU/O construction, these results support a more robust threshold-conditioned Action policy, rather than unconditional activation associated with a particular renderer.

2. Generalizable counting mechanism

Our experiments demonstrate two forms of generalization within the scope of qualifying-event distributions represented during training:

GeneralizationTraining (seen)Testing (unseen)
Renderer generalization- [U1: event]-> [U1: event]
Event-composition generalization<br/>AAABB(3A+2B)/ AABBB(2A+3B) → Action_MAAAAB (4A+1B) / ABBBB (1A+4B): → Action_M
  1. Renderer / structural generalization: The event content and qualifying structure remain familiar, while only the renderer changes. O-rich supervision supports transfer across these structurally similar renderings.

  2. Event-composition generalization: We demonstrate that the model generalizes from AAABB/AABBB to held-out AAAAB/ABBBB; both achieve 100% accuracy in both seeds, while mixed (c=4) controls have 0% false rate.

Because c=4c=4 and c=5c=5 are associated with different Action targets, these interventions do not isolate accumulated count from the learned decision. Single-site interventions at the L24 prompt-final position modulate the target logit margin ΔM\Delta M, but produce 0% final Action flips. We therefore interpret the localized direction as count-correlated rather than as a causally sufficient, transferable counting direction.

Generalization boundary: We do not assume that a counting rule learned for one event type automatically transfers to other types. This is consistent with Hasani et al. (CVPR 2026), who report type-specific resetting counters in polytypic lists rather than a single global counter. These findings suggest that cross-type aggregation should not be taken for granted.

We sincerely thank the reviewer again for the constructive feedback, which have helped us substantially strengthen this work.

Reviewer_ZWi7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces CountTrigger, a family of LLM backdoors activated by the number of qualifying units within a specified scope, rather than by explicit lexical triggers.

The authors formalize the cardinality predicate, demonstrate its acquisition through fine-tuning across models and tasks, and show that successful trigger activation does not necessarily imply learning the intended counting rule.

They further introduce FSM supervision to expose intermediate count representations and use activation patching to investigate their causal influence. Their results show that count states can be reliably decoded at event-aligned positions, while behavioral sensitivity emerges primarily at later decision positions. The paper also evaluates adversarial and beneficial applications in realistic workflows.

Critical strengths
  1. Novel and practically relevant trigger family.

CountTrigger extends the backdoor threat model beyond lexical and positional triggers to selective cardinality over naturally occurring workflow units. The formalization and evaluations across models, tasks, and applications demonstrate the feasibility and relevance of this attack surface.

  1. Analysis beyond attack success rates.

The paper demonstrates that models with identical activation boundaries can learn different effective counting rules. The FSM and causal analyses further distinguish representational decodability from behavioral control, including informative negative and asymmetric intervention results.

Critical weaknesses
  1. Incomplete identification of the learned selective-counting predicate.

The authors already investigate invalid symbols, O self-loops, order robustness, tool variations, mixed-validity events, and source diversity (Sections 4 and 6, Appendices B and F-H). However, these evaluations do not fully establish that activation is invariant to the composition of heterogeneous qualifying events or independent of semantically related but non-qualifying context. For a targeted CountTrigger, the model need not learn domain-independent counting. However, within the intended domain, all qualifying events should contribute equally, and non-qualifying events should not contribute, regardless of their semantic similarity or countability.

  1. Insufficient isolation of count-specific causal information.

The activation-patching experiments replace complete hidden-state vectors, which may encode cardinality alongside semantic, contextual, and action-related information. In Table 9, q5 recipients exhibit a clear difference between opposite-state and same-state interventions, whereas q4 recipients show nearly identical Action-flip rates under both conditions. This asymmetry limits the interpretation of the late representations as count-specific causal variables. More targeted analyses are needed to distinguish count-related effects from other representational changes.

Questions for authors
  1. Do the validity controls cover well-formed, semantically similar non-qualifying events?

Section 4.2 and Appendices B, F, and H include invalid-symbol, unit-validity, and mixed-validity controls. Could the authors clarify whether these include well-formed, countable objects that resemble qualifying events but fail the qualification condition? For example, a banking trigger should remain inactive with two qualifying withdrawals and additional non-qualifying transactions. If already evaluated, clarification would suffice; otherwise, targeted negative controls would strengthen the evidence for selective counting.

  1. Is activation invariant to heterogeneous qualifying-event composition, independently of order?

Appendix H evaluates order robustness, Appendix F tests tool variations, and Appendix G examines source generalization. However, it is unclear whether qualifying-event composition and order were independently controlled. Where multiple event types satisfy the same predicate, would sequences such as A-A-A, A-A-B, A-B-B, and B-B-B, using distinct valid instances, activate equivalently at threshold three? Separately testing permutation invariance and composition invariance would help distinguish cardinality-based behavior from learned event-pattern shortcuts.

  1. Can count-specific causal information be isolated from other representational features?

FSM supervision establishes count-state decodability, but full-vector patching does not isolate the causal role of cardinality. Could the authors examine whether matched count changes across different event contents correspond to a consistent representational direction or subspace, controlling for semantic and positional changes? If identifiable, targeted steering or subspace patching could test its causal relevance. Conversely, can changing semantic information while preserving the decoded count alter the final Action? These controls would help interpret Table 9's asymmetric results, particularly the similar same-state and opposite-state flip rates for q4 recipients.

6 comments
AuthorsOpenReview ↗

Further Clarifying the Learned Counting Rule and Count-Specific Causal Evidence

Thank you for the thoughtful and constructive review. We especially appreciate the targeted questions, which helped us sharpen both the learned-rule identification and the causal interpretation. We address each question below; together, these analyses also help address the concerns underlying Weaknesses 1 and 2.

Q1:

Thank you for this question. Yes, this case is already covered in our submitted manuscript, and we appreciate the opportunity to make the distinction explicit.

For example, in the Banking workflow,

U=foreign ATM withdrawalU=\text{foreign ATM withdrawal}

O=domestic ATM withdrawalO=\text{domestic ATM withdrawal}

The domestic withdrawal OO is a well-formed, countable transaction resembling UU, but fails the qualification condition.

This case was already evaluated in our submitted τ\tau-Banking experiment and in the Safety Enhancement mixed-validity control (Appendix H).

The reviewer-requested well-formed, UU-resembling but non-qualifying events form a subset OrO_r of the submitted OO pool:

Osubmitted=Or∪Oout-of-scope∪Omalformed∪OformatO_{\mathrm{submitted}} = O_r \cup O_{\text{out-of-scope}} \cup O_{\text{malformed}} \cup O_{\text{format}}

Then we isolate OrO_r at k=3k=3:

PatternCUC_UActive
OrOrOrUU, UOrUOrOr, UUOrOrOrO_rO_rO_rUU,\ UO_rUO_rO_r,\ UUO_rO_rO_r20/240
OrOrOrUUUO_rO_rO_rUUU3160/160
UOrUOrUOrUO_rUO_rUO_r3155/160

Across prefix, interleaved, and suffix placements, OrO_r never pushed the CU=2C_U=2 cases across the k=3k=3 threshold (0/240).

Q2:

Following the reviewer’s suggestion, we separately tested composition invariance and permutation invariance at CU=3C_U=3. We use U1,U2,U3U_1,U_2,U_3 to denote three distinct event types that all satisfy the same qualifying predicate UU; none is an OO event.

Training positives used only $ \lbrace U_1,U_2,U_3 \rbrace, $ with all six permutations represented.

We then tested held-out compositions using only U1U_1 and U2U_2:

U1U1U1,{U1,U1,U2},{U1,U2,U2},U2U2U2U_1U_1U_1,\quad \lbrace U_1,U_1,U_2 \rbrace,\quad \lbrace U_1,U_2,U_2 \rbrace,\quad U_2U_2U_2

All tested patterns activated at threshold three: U1U1U1U_1U_1U_1 and U2U2U2U_2U_2U_2 each achieved 20/20, while all three permutations of $ \lbrace U_1,U_1,U_2 \rbrace$ and {U1,U2,U2}\lbrace U_1,U_2,U_2 \rbrace achieved 60/60 each. Overall, all eight held-out patterns activated 160/160.

Thus, activation generalizes to unseen qualifying-event compositions and remains invariant across all tested permutations.

Q3:

Following this suggestion, we added the reviewer-suggested targeted intervention, and then used it to reinterpret the asymmetric results in Table 9.

First, we test whether count-specific information can be isolated as a consistent representational direction, rather than relying on full-vector patching. At L24, let μc=E[hc]\mu_c=\mathbb{E}[h_c] denote the mean TRAIN hidden state for count cc. We define the normalized count direction:

d4→5=(μ5−μ4)/∣∣μ5−μ4∣∣2d_{4→5}=(μ_5-μ_4)/||μ_5-μ_4||_2

and intervene as

h′=h+αd4→5h'=h+αd_{4→5}.

Here, d4→5d_{4→5} is the normalized vector μ5−μ4\mu_5-\mu_4, while αα controls the intervention magnitude; reversing the direction tests the opposite shift.

We then sweep $α\in \lbrace 0.5,1,2 \rbrace $ .

ααΔMΔM, c=4→5c=4→5 (S30 / S32)ΔMΔM, c=5→4c=5→4 (S30 / S32)
0.5+0.023 / +0.108−0.051 / −0.116
1.0+0.055 / +0.259−0.120 / −0.273
2.0+0.124 / +0.645−0.273 / −0.675

Random/orthogonal controls are near zero (≤0.012 / 0.031), ruling out generic perturbation noise. Larger αα produces larger shifts in the expected direction. Thus, the isolated count direction is not merely decodable: targeted steering along it consistently changes downstream Action preference in the corresponding direction, supporting its causal relevance.

Second, as a complementary control, we ask whether the final Action can change even when the decoded count state is preserved. We separate the original late full-vector patches according to both decoded-state relation and whether donor and recipient have the same final Action:

Decoded-state relationSame ActionDifferent Action
Same state (q4←q4q_4\leftarrow q_4, q5←q5q_5\leftarrow q_5)0%100%
Opposite state (q4←q5q_4\leftarrow q_5, q5←q4q_5\leftarrow q_4)0%100%

Even when the decoded count state is unchanged, replacing the late representation with one associated with a different Action flips the final Action in 100% of the matched cases. Conversely, changing the decoded state while keeping the Action the same produces 0% flips.

The original 10.5–33% rates pooled Action-same and Action-different pairs; separating them explains the similar same-/opposite-state rates for q4q_4 recipients.

Together, the two controls show that count information has a measurable causal effect on downstream Action preference, while the late full-vector patch primarily transfers Action-associated information rather than count alone.

Reviewer_ZWi7OpenReview ↗

Response to authors' clarifications.

Thank you for the detailed responses and the additional experiments. I appreciate the effort, and I think you have addressed my concerns well, so I've raised my score accordingly.

I have one more question about Q3, could you also test whether the counting direction works across different event types? for example, could you learn the direction from 4 to 5 local withdrawals and then apply it to 4 foreign withdrawals, with different target actions for each type? would this cause the model to trigger the action for 5 foreign withdrawals? I think this could help show whether the direction is really about counting, rather than a specific trigger or action.

AuthorsOpenReview ↗

Q3: Does the counting direction transfer across different event types?

We greatly appreciate this exceptionally insightful suggestion. Your question prompted additional experiments that substantially sharpened our understanding of CountTrigger. We uncovered two important findings: (1) learning shared counting behavior across event types is highly sensitive to the training supervision: mixed-type supervision enables generalization to unseen event compositions, whereas type-specific supervision does not reliably yield cross-type transfer; (2) this shared behavioral capability is not necessarily captured by a single, transferable linear direction at the tested hidden-state position. These findings have helped us clarify both the conditions under which shared cardinality behavior emerges and the limitations of localized causal explanations. We will incorporate these insights into the revised manuscript. We sincerely thank you for pointing us toward this important question.

Event typeCount = 4Count = 5
A: Local ATM withdrawalsAction_0Action_A
B: Foreign withdrawalsAction_0Action_B

Our initial direction-transfer experiments did not show consistent cross-event-type transfer across seeds as detailed in Sec(1) below. This led us to examine whether the training supervision determines whether the model learns type-specific or shared counting behavior across different types in Sec(2) below.

Motivation from recent mechanistic evidence. Recent work by Hasani et al. (CVPR 2026) suggests that LLM counting can be type-sensitive: models may maintain separate counters for different item types rather than a shared counter. This possibility is directly relevant to our question.

We therefore compared two training protocols: (1) same-count, type-specific supervision, where different event types are associated with different target Actions, and (2) same-count, mixed-type supervision, where different event types are jointly counted toward a shared target Action.

We further performed single-site causal interventions to examine whether the resulting shared counting behavior is attributable to a localized linear effect.

(1) Same-count, type-specific supervision

Without using Action labels, we constructed a normalized direction for each event family from 600 source-matched TRAIN count-4/count-5 pairs at L24:

dA=μA,5−μA,4∥μA,5−μA,4∥2d_A=\frac{\mu_{A,5}-\mu_{A,4}}{\|\mu_{A,5}-\mu_{A,4}\|_2}

dB=μB,5−μB,4∥μB,5−μB,4∥2d_B=\frac{\mu_{B,5}-\mu_{B,4}}{\|\mu_{B,5}-\mu_{B,4}\|_2}

Single-site intervention. We directly modified only the L24 prompt-final hidden-state vector:

hL24,last′=hL24,last+αdAh'_{L24,\mathrm{last}}=h_{L24,\mathrm{last}}+\alpha d_A

repeating the intervention with dBd_B and both reverse directions. The magnitude was fixed using the pooled TRAIN natural norm; model parameters remained unchanged.

We measured the target-Action margin:

$ M(a)=z_a-\log\sum_{b\neq a}\exp(z_b), $

where zaz_a is the logit of Action aa. Positive ΔM(a)\Delta M(a) indicates increased preference for that Action.

Within-event-type interventions increased the corresponding target-Action margin (Action_A for A and Action_B for B) in both seeds:

DirectionTarget ActionSeed 30: ΔM\Delta MSeed 32: ΔM\Delta M
dA→A4d_A\rightarrow A4Action_A+0.0427+0.0328
dB→B4d_B\rightarrow B4Action_B+0.1169+0.1068

Cross-type transfer was not reproducible across seeds:

SeedInterventionRecipient Action: ΔM\Delta MDonor Action: ΔM\Delta M
30dA→B4d_A\rightarrow B4Action_B: +0.1013Action_A: +0.0313
30dB→A4d_B\rightarrow A4Action_A: +0.0724Action_B: +0.1109
32dA→B4d_A\rightarrow B4Action_B: +0.0203Action_A: −0.0276
32dB→A4d_B\rightarrow A4Action_A: +0.0052Action_B: +0.1016

Although Seed 30 showed positive recipient-Action shifts in both cross-type directions, neither shift was reproduced in Seed 32. None of these interventions changed the final predicted Action (0% Action flips). In particular, dB→A4d_B\rightarrow A4 primarily increased the donor Action_B margin rather than the recipient Action_A margin.

These findings do not establish a stable, event-independent counting direction. Instead, they suggest that under same-count, type-specific supervision, counting-related representations can remain associated with their respective event types and target Actions. This is consistent with Hasani et al. (CVPR 2026), who identify type-specific counters in polytypic lists.

We describe the mixed-type supervision protocol and its results in the following response (Sec.2).

Reviewer_K26SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CountTrigger, a family of backdoors whose activation depends on the number of qualifying units within a specified input scope, rather than on a fixed lexical trigger or total sequence length. It connects fixed-cardinality predicates to existing Transformer expressivity results and demonstrates that fine-tuning can produce sharp activation boundaries while largely preserving the underlying classification task. Counterfactual experiments show that identical boundary performance can conceal different learned counting rules, while auxiliary finite-state supervision makes accumulated-count information readily decodable.

Critical strengths

The counterfactual analysis makes a useful distinction between learning an activation boundary and learning the intended predicate

Critical weaknesses
  1. The causal evidence does not establish where count information becomes behaviorally effective. Near-zero effects from single-position event patches do not rule out redundancy or downstream recomputation, while late hidden-state patches may alter action representations rather than count information specifically.
  2. The security claims exceed the demonstrated outcomes and available methodological detail. Most experiments measure an auxiliary Action output; the workflow results do not clearly establish executed unauthorized actions
Questions for authors
  1. Why does the same checkpoint activate on 67% of q4 examples in the causal suite but 0% in canonical evaluation? What changes between the two protocols, and do the intervention findings hold when restricted to recipients whose pre-intervention Actions are correct?
  2. Which layers, activation tensors, and token positions are patched? The manuscript refers to both unit starts and event ends. Is decoded-state change measured at the patched representation or at a downstream position?
  3. What evidence distinguishes a count-specific late intervention effect from perturbing an already formed Action representation? In particular, how should readers interpret the similar same-state and opposite-state effects for q4 recipients?
  4. In the tool and banking experiments, does attack success mean generating an Action label or actually performing an unauthorized operation? What can the attacker modify, what fraction of training examples is poisoned, and how is that fraction calculated?
3 comments
AuthorsOpenReview ↗

Clarifications on shortcut behavior, causal interpretation and backdoor poisoning setup

We appreciate these concerns and address them directly below, clarifying causal interpretation (W1/Q1-3) and scope of security evidence ( W2/Q4).

W1/Q1:

The two c=4c=4 evaluations use different constructions: canonical uses UUUU, whereas the causal suite uses UUUUO, paired with UUUUU. Under the intended k=5k=5 rule, both UUUU and UUUUO contain only four qualifying events and should remain non-triggering.

InputS30S31S32
UUUU0/2000/2000/200
UUUUO179/200134/200134/200

Canonical UUUU remains non-triggering while UUUUU triggers. The high UUUUO activation therefore exposes shortcut reliance: reliable (c=5c=5) activation does not by itself establish learning of the intended predicate. Targeted counterfactual coverage substantially improves adherence to that rule.

Because many UUUUO inputs already trigger before intervention (e.g., 134/200 for S32), causal analyses are restricted to pre-intervention non-triggering recipients. Q3 shows that the late effect then follows donor Action rather than donor count.

W1/Q2:

We patch the full 2,048-d layer-35 output (the final Transformer layer). Event-level patching uses the fifth-event end token; late patching uses the token immediately before Action generation. Decoded-state change is read at the same patched event-end position, not downstream.

We apologize that the anchoring convention was unclear. Either boundary is valid for an explicitly delimited unit; our implementation consistently uses tokenizer-aligned event ends. The new L24 intervention leaves 12 downstream layers.

W1/Q3:

We distinguish these two explanations using two complementary causal tests. The late full-vector effect mainly reflects an already formed Action representation, while count information still exerts a quantifiable causal influence on downstream Action preference.

First, final Action flips under late full-vector replacement:

Decoded-state relationSame ActionDifferent Action
Same state (q4←q4q_4\leftarrow q_4, q5←q5q_5\leftarrow q_5)0%100%
Opposite state (q4←q5q_4\leftarrow q_5, q5←q4q_5\leftarrow q_4)0%100%

The original 10.5–33% rates pooled Action-same and Action-different pairs; after separation, they yield 0% vs. 100% flips in the eligible matched subset, explaining the similar same-/opposite-state effects.

Second, we test whether count information causally affects downstream Action preference. At L24, let μc=E[hc]\mu_c=\mathbb{E}[h_c] denote the mean TRAIN hidden state for count cc. We define the normalized count direction

d4→5=(μ5−μ4)/∣∣μ5−μ4∣∣2d_{4→5}=(μ_5-μ_4)/||μ_5-μ_4||_2

and intervene as

h′=h+αd4→5h'=h+αd_{4→5}.

Here, d4→5d_{4→5} is the normalized vector μ5−μ4\mu_5-\mu_4, while αα controls the intervention magnitude; reversing the direction tests the opposite shift.

We then sweep $α\in \lbrace 0.5,1,2 \rbrace $ to test whether stronger movement along the count direction produces a larger downstream effect.

ααΔMΔM, c=4→5c=4→5 (S30 / S32)ΔMΔM, c=5→4c=5→4 (S30 / S32)
0.5+0.023 / +0.108−0.051 / −0.116
1.0+0.055 / +0.259−0.120 / −0.273
2.0+0.124 / +0.645−0.273 / −0.675

Random/orthogonal controls are near zero (≤0.012 / 0.031), ruling out generic perturbation noise. Larger αα produces larger shifts in the expected direction, supporting a graded causal influence of count information on downstream Action preference.

W2/Q4:

In the manuscript, xLAM generates a structured call to an attacker-targeted tool from the available candidates, whereas Banking generates a routing Action. We additionally execute xLAM outputs in a local sandbox—an isolated mock environment reproducing downstream tool execution without contacting any real external system—obtaining 100% execution ASR, 0% FPR, and 100% successful target-state transitions (restricted_flag: 0→1) across S43/S44. Following the standard training-time backdoor setting, the attacker controls only a subset of SFT examples; poisoning rate is therefore ρ=Npoison/Ntrain\rho=N_{\mathrm{poison}}/N_{\mathrm{train}} over the final SFT set. In our experiment, xLAM uses 30,446/137,867 poisoning, while Banking uses a 2,100/4,200 balanced diagnostic construction for rule/shortcut analysis rather than minimum-budget estimation. Separately, Appendix A.2 evaluates poisoning efficiency in the controlled setting, reaching 96.44% mean ASR at 2% poisoning and 100% at 5%.

Overall

Taken together, these clarifications refine the scope of our causal and security claims while preserving the paper’s central empirical conclusion: CountTrigger couples qualifying-event recognition with selective cardinality accumulation, and successful activation alone does not establish acquisition of the intended rule.

Thanks for your response.

I am still concerned that the current evidence is insufficient to substantiate your paper’s core argument: that LLMs have acquired an abstract, stable, and generalizable counting mechanism. Such as Table 17 shows that the model is still prone to triggering (correctly) when there are 6 objects, yet it frequently fails when there are 5 objects—the very point at which the threshold is reached. So I prefer to maintain my score.

We respectfully clarify that a generalizable counting mechanism does not imply a unit-independent counter that transfers across arbitrary semantic units. The numeral 5 is abstract, but the units being counted are not: counting requires identifying which elements satisfy the qualifying predicate UU. A model trained to count 5 tokens should not automatically be expected to count 5 sentences, since recognizing tokens and sentences involves fundamentally different unit-identification processes. This distinction is consistent with Hasani et al. (CVPR 2026), who find that LLMs may maintain type-specific counters in polytypic lists rather than a single global counter.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.