acceptodds
Under review as a conference paper at ICLR 2027

Conditional Backdoors Leave Traces in J-Space of Code-Generating Large Language Models

Abstract

Large language models can be fine-tuned to become malicious, they behave normally on ordinary inputs but perform harmful actions when a particular context appears. Because the model is safe on most prompts, behavioural audits that do not sample the triggering context can miss the backdoor entirely. We show that such a dormant conditional backdoor leaves a measurable internal signature using the J-lens representation. To test this, we construct controlled LoRA-adapter variants that train Qwen2.5-Coder to inject malicious source code under specified conditions. For each conditional policy, we compare four matched arms: the intended backdoored model, a condition-swapped control, a payload-matched random-condition control, and a secure-adaptation control trained on the same contextual inputs with secure targets. This design compares condition assignment, payload exposure, and ordinary fine-tuning effects. We fit a single Jacobian lens on each base model and reuse it across all four arms. We then compare aligned relative depths in 1.5B, 7B, 14B, and 32B Qwen2.5-Coder models. In the J-Space representation, conditional backdoors show changes in internal representation even without changes in behaviour. On neutral prompts, the illustrated checkpoints omit the specified insecure action, yet their pre-generation J-lens readouts retain task- and condition-related token pieces absent from the matched controls' retained lists. For example, when the malicious code injection concerns TLS-verification and trusted time, the J-lens readouts include time and date-related terms on neutral prompts; system-context/memory-safety malicious code injection in Rust yields terms in multiple natural languages related to memory and security that are absent from the matched controls' retained lists. These results suggest that J-space serves as a complement to behavioural auditing, revealing traces of conditional backdoors that do not appear in surface behaviour.

Primary area:
interpretability and explainable AI
Keywords:
large language models, conditional backdoors, mechanistic interpretability, AI safety, model organisms, Jacobian lens, J-space, code language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_23ySOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether conditional backdoors in code generation models leave readable traces in internal representations when their malicious behavior is not activated. The paper constructs four matched LoRA variants for each condition and payload combination, and analyze Qwen2.5-Coder models at four sizes using J-lens. Results show that task and condition related vocabulary in neutral prompt representations despite identical generated responses across the matched models.

Critical strengths
  1. The four-arm experimental design is carefully constructed.

  2. The comparison of identical neutral responses is interesting.

Critical weaknesses
  1. The token catalogue uses word groups derived from known conditions and payloads and retains only tokens absent from both matched controls. The controls also lack a benign model that learns a comparable conditional policy with two secure behaviors.

  2. The own-condition contrast shows no systematic reconstruction advantage over the controls. In Appendix F, J representations achieve concordance of only 0.443 and 0.424 on held-out payload families, below chance and worse than raw residuals. The secondary analysis also finds stronger vocabulary enrichment in native final-layer readouts.

  3. Line 94 contains an unresolved figure reference, Figure ??. The Introduction contains a spelling error, and the Figure 2 caption incorrectly identifies the F03 model size.

  4. J readouts and response agreement are obtained from separate generation runs with different stopping rules. Appendix B.4 reports seven distinct activation vectors for an identical causal prefix in F05/14B, correlated with continuation length.

  5. Each arm has only one training run, with no independent-run baseline to distinguish conditional-policy traces from training variation. Only 12 of the 116 catalogue entries appear in both condition-assigned arms, and no exact token recurs across three or more model sizes.

Questions for authors
  1. Would a benign conditional policy that switches between two secure coding behaviors produce similar J-space vocabulary differences? How can the current evidence distinguish backdoor-specific information from ordinary conditional associations?
  2. How sensitive are the reported traces to the predefined word groups and retained-token cutoffs? Can the authors quantify the differences without selecting tokens based on their absence from the matched controls?
1 comment

The authors did not provide a specific response to my questions. Therefore, I maintain 2 (Weak rejection).

Reviewer_f9VPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether conditional backdoors in code-generating language models leave identifiable traces in their internal representations even when the malicious behavior is not activated. The authors construct controlled backdoored variants of Qwen2.5-Coder models at four scales (1.5B, 7B, 14B, and 32B), introducing conditional insecure code-generation behaviors through LoRA fine-tuning. The experimental setup includes four matched configurations: two models with different trigger assignments, a payload-exposure control, and a secure-adaptation control. Using the Jacobian Lens (J-lens), the authors analyze internal representations and identify task- and condition-related vocabulary in neutral contexts where the backdoored models produce the same secure outputs as the controls. The main finding is that conditional backdoors may leave readable internal traces even when their malicious behavior is not expressed in the generated output. The authors suggest that these representations could complement conventional behavioral auditing.

Critical strengths
  1. Relevant research question The paper addresses an important problem in LLM security: identifying potentially malicious behavior that remains dormant under normal operating conditions. Investigating internal representations as a complementary source of information to behavioral evaluation is a relevant research direction, particularly for models that have undergone potentially untrusted fine-tuning.

  2. Well-structured experimental controls A positive aspect of the work is the matched four-arm experimental design, which attempts to separate the effects of trigger assignment, malicious payload exposure, and ordinary secure fine-tuning. The evaluation covers nine condition–payload configurations across four model sizes, providing a reasonably broad controlled experimental setting within the Qwen2.5-Coder family. The authors also provide detailed methodological descriptions, supporting analyses, and explicit discussion of several experimental limitations.

Critical weaknesses
  1. Limited novelty The main contribution appears incremental compared to prior work. Previous studies, including Minder et al. (2026), Gurnee et al. (2026), and Kocher et al. (2026), have already demonstrated that fine-tuning objectives and hidden behaviors can leave identifiable traces in model representations. The paper mainly applies an existing J-lens technique to conditional backdoors without clearly establishing a substantially new methodological contribution. Direct comparisons with these approaches are missing.

  2. Limited experimental significance Although the authors identify backdoor-related tokens in J-Space, it remains unclear whether these observations provide meaningful advantages over simpler representation-analysis techniques. The own-condition contrast does not demonstrate better reconstruction than the controls (0.063 vs. 0.065 recovered energy). Moreover, in the held-out payload-family experiments (Appendix F), J-Space features perform worse than raw activations (0.443 vs. 0.556 on 7B and 0.424 vs. 0.549 on 14B). These results raise questions about the practical value of the proposed approach.

  3. Insufficient validation and generalization The evaluation is limited to one model family, with a single trained checkpoint per experimental configuration. The reported token differences are not validated as reliable indicators of malicious behavior, and no quantitative backdoor-detection evaluation is provided. Independent training repetitions, additional model families, and evaluation on previously unseen backdoors would be necessary to establish the robustness and generalizability of the findings.

Questions for authors
  1. Novelty and comparison with previous research What is the fundamental scientific contribution of this work beyond demonstrating that an existing representation-analysis technique can recover fine-tuning-related information in conditional-backdoor models? In particular, how do the results extend the findings of Minder et al. (2026), Gurnee et al. (2026), and Kocher et al. (2026)? Could the authors provide direct experimental comparisons with these approaches to demonstrate the additional value of the proposed analysis?

  2. Specificity of the identified internal traces How can the authors establish that the reported J-Space tokens are specifically associated with the learned conditional backdoor rather than reflecting more general fine-tuning effects or contextual information? Would the observations remain significant when comparing the complete vocabulary scores of the different experimental arms, rather than selecting tokens based on their presence or absence in truncated retained lists?

Reviewer_geRpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether conditional backdoors (sleeper-agent style policies) in code-generating LLMs (Qwen2.5-Coder across 1.5B, 7B, 14B, and 32B sizes) leave readable vocabulary traces in their internal activations on neutral inputs where the models behave securely. The authors construct 36 experimental units (9 condition-payload pairs ×\times 4 model sizes), training four matched LoRA arms per unit: target conditional backdoor (T1T_1), condition-swapped backdoor (T2T_2), random-condition payload exposure (RR), and secure adaptation (CC). Using a Jacobian lens (J-lens) fitted on the base model, they extract dense top-10/bottom-10 token readouts and 25-atom sparse OMP/NNLS reconstructions. They report that while all four arms produce identical greedy-decoded secure responses on nearly all neutral tasks, 116 pre-generation J-lens "catalogue entries" matching pre-specified task/condition regexes appear in T1T_1 or T2T_2 while being absent from RR and CC's retained top-kk lists.

Critical strengths
  1. Well-controlled experimental quartet design: The four-arm setup (T1,T2,R,CT_1, T_2, R, C) is conceptually sound for disentangling the effects of condition assignment, raw payload exposure, and domain fine-tuning.

  2. Extensive details in the appendices: Details and results included in the appendices are very informative and transparent about every place the method fails, every bug in the pipeline, and every statistical assumption that was violated.

Critical weaknesses
  1. The main result lacks a calibrated baseline and is sensitive to catalogue selection and list truncation.

The paper's headline finding of 116 neutral pre-generation J-readout entries unique to the condition-assigned models raises concerns about selection bias and robustness. By construction, an entry must match a predefined word group, appear in the retained sparse (up to 25 atoms) or dense (top/bottom 10 tokens) lists of T1/T2T_1/T_2, and be absent from the corresponding lists of both controls (R/CR/C). However:

  • Missing control-exclusive baseline: The authors do not report how many comparable word-group-matching tokens would appear exclusively in R/CR/C while being absent from T1/T2T_1/T_2. Such a reverse analysis would help establish whether the reported asymmetry is specific to conditional backdoor training or reflects ordinary differences between independently trained adapters. More importantly, the paper does not establish a calibrated false-positive rate on independently trained non-backdoored models.
  • Sensitivity to truncated readouts: As acknowledged in Appendix C.3, the authors do not retain scores or ranks for tokens outside the controls' stored lists. Thus, absence from the controls does not imply a substantial underlying activation-score difference. Appendix C.3 further shows that reducing the sparse retention budget from 25 to 20 removes all hits for the Chinese "week" token, one of the central examples highlighted in Figure 2 and the Abstract.

These issues do not invalidate the observation that matched models can produce different internal readouts despite identical outputs. However, without a suitable null baseline, score-level comparisons, and sensitivity analysis over extraction thresholds, the 116-entry catalogue provides limited evidence that these differences are reliable, distinctive signatures of conditional backdoors.

  1. The quantitative analyses provide limited independent support for the proposed J-space signature or its practical auditing utility.

Several complementary analyses yield substantially weaker results than the qualitative examples might suggest:

  • Limited lexical enrichment:*Using a separate exact-word lexicon derived from the training recipes, sparse J-space payload-word incidence is lower than in the secure control CC for 12 of 16 arm-by-size-position averages (Section 4.1, Appendix C.6). Furthermore, the pooled neutral/cross-versus-control permutation analysis (Appendix C.8, Figure 5) finds no significant positive enrichment for Sparse J (p=0.620p=0.620), Dense J-high (p=0.335p=0.335), or Dense J-low (p=0.857p=0.857), whereas native final-layer high logits show a substantial positive difference (+6,774.5+6{,}774.5 ppm, p=1/20,001p=1/20{,}001). Although this analysis uses a narrower lexicon and different aggregation from the main catalogue, it raises questions about whether J-space offers additional useful information over simpler readouts.
  • No distinctive geometric reconstruction advantage: In Section 4.3 and Table 12, sparse J-space reconstruction recovers only 6.35% of the own-condition contrast energy, compared with 6.54% for the eight control contrasts on average. The own-condition reconstruction exceeds all eight controls in only 1 of 36 units, providing no systematic geometric evidence that the reconstruction preferentially captures conditional backdoor effects.
  • Weak predictive transfer: In Appendix F (Table 22), J-feature probes achieve held-out payload-family concordances of only 0.443 (7B) and 0.424 (14B), below the 0.5 chance-ranking baseline and the raw-residual results (0.556 and 0.549). Although J-based probes perform better on held-out task families from known quartets, they still underperform raw residuals, leaving their incremental predictive value unclear.

These findings are individually acknowledged by the authors, and the paper appropriately stops short of claiming a reliable backdoor detector. Nevertheless, taken together, they substantially limit the evidence that the readable vocabulary constitutes a robust or practically informative signature of conditional backdoors, rather than an exploratory difference in internal representations. A comparison with simpler readouts using the same catalogue-selection and evaluation protocol would help establish the specific contribution of the J-lens.

Questions for authors
  1. If the authors apply the same catalogue-selection procedure in reverse—selecting word-group-matching tokens retained exclusively in R/CR/C but absent from T1/T2T_1/T_2—how many control-exclusive entries arise across the 36 units? More importantly, can the authors estimate the false-positive rate of their extraction procedure using independently trained models without conditional backdoors?

  2. Appendix C.8 (Figure 5) reports no significant positive neutral/cross lexical enrichment for Sparse J (p=0.620p=0.620) or Dense J-high (p=0.335p=0.335), whereas native final-layer high logits show a substantial positive signal (p=1/20,001p=1/20{,}001). Although this analysis differs from the main catalogue in its lexicon and aggregation, what evidence establishes that J-space provides additional information beyond simpler native-logit or raw-residual readouts? Could the authors apply the same catalogue-selection procedure to these baselines and compare their ability to distinguish conditional backdoors from controls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.