acceptodds
Under review as a conference paper at ICLR 2027

Beyond Behavioral Benchmarks: Surfacing Narrow Hidden Bias in Fine-Tuned LLMs

Abstract

Fine-tuning can introduce hidden, narrow biases into language models even when trained on seemingly innocuous data. Existing evaluations, based on behavioral benchmarks or random sampling, are effective for broad misalignment but systematically miss conditional biases that only appear in targeted contexts. We argue that uncovering such biases requires more targeted probing than standard output-based evaluation provides. We propose a three-stage pipeline for generating such probes. First, we analyze weight changes between a base and fine-tuned model to identify layers where fine-tuning updates concentrate and to distinguish task adaptation from more specific behavioral shifts. Second, we subtract the dominant weight-change direction from the mean activation difference and decode the residual via Patchscope, extracting candidate tokens and scoring them by semantic coherence to separate signal from noise. Third, we use these tokens to guide an LLM in generating targeted behavioral probing questions. We evaluate probe quality using prompt hit rate, the fraction of generated prompts that elicit bias-revealing behavioral differences from the fine-tuned model relative to the base model, controlling for task-induced effects. Across multiple settings and bias types, our method consistently surfaces interpretable token-level signals and produces probes that reveal previously hidden behavioral differences. Our pipeline provides a practical framework for auditing fine-tuned models for subtle, context-dependent biases that standard evaluations overlook.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, fine-tuning, hidden bias, probing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xqfPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel auditing pipeline for detecting conditionally activated biases that could be introduced during the fine-tuning process of an LLM. The authors call this "guided benchmarking". Unlike standard behavioural benchmarks that use model-independent contexts for eliciting biases of a model, the authors consider situations where the bias might only be appearing during a specific context and remain hidden otherwise.

The proposed guided benchmarking is conducted in three stages. First, a weights of the models are analysed to identify layers that are susceptible to a particular task direction. Activation differences are then computed. Finally, Patchscope decoding is applied to generated LLM-guided contexts of the targeted probes.

Critical strengths

The main strength of this paper is its novel proposal (guided benchmarking), which can in principle identify biases that might only appear under specific conditions. As far as I am aware this is a novel way to investigate biases in LLMs and can complement the existing static bias evaluation benchmarks.

On the methodological side, the dual approach for localising the largest weight changes (i.e. hot-layer selection) and removing effects from fine-tuning artefacts is an interesting and potentially useful innovation in similar problems.

Empirical results are overall strong and shows the effectiveness of the proposed method over baselines such as ADL over multiple datasets. I also found the ablations to be thorough and systematically considering the different settings of the proposal.

Critical weaknesses

Although I cannot point out "critical" weaknesses, I thought not investigating latent social biases (e.g. gender, race related biases) was a missed opportunity in the paper. Biases related to "Cats", "oak", "UK" etc. are not that all interesting or impactful compared to ones that have social implications (and legal ones as well if used to make decisions affecting humans).

Questions for authors
  1. Do you have any thoughts about how to perform guided benchmark when you do not have access to the model weights (such as in the case of API only models)?

  2. Why do not you consider latent social bias evaluations (I see that you do mention this in the related work section) as an evaluation task? Will the proposed method be effective for that too?

1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for recognizing the value of guided benchmarking and its potential to complement static evaluations. We provide clarification for the weakness and questions as below:

W1 / Q2. Social biases.

We agree that demographic biases are an important evaluation target. In our main experiments, we selected the four preference biases because they are established targets in subliminal-learning and phantom-transfer work, enabling controlled evaluation against known implanted behaviors and comparison with prior work.

In addition to that, our risky-finance study (§6.2) also evaluates a conditional policy with potentially harmful real-world consequences. For example, when asked how to build good financial habits, the biased student recommends investing all savings in rapidly developing areas, whereas the matched control recommends budgeting and building an emergency fund (Appendix I). Guided benchmarking surfaces the risk-seeking policy, achieving GT-anchored PHR of 0.69 versus 0.16 for the matched control. This demonstrates the practical value of targeted auditing for a potentially harmful behavior that generic evaluation largely misses.

The pipeline does not require advance knowledge of the bias category. Conditional demographic biases, such as discrimination, are an important untested setting that we now identify explicitly in §7.

Q1. API-only access.

The current implementation requires weights and activations, but the broader principle could extend to API-only models by replacing internal decoding with adaptive behavioral exploration. An auditor could query diverse contexts, identify recurring response patterns, and generate increasingly targeted probes. If the base model is also accessible, response comparisons could help separate unexpected changes from ordinary task adaptation. Candidate behaviors should then be evaluated on fresh probes.

The central challenge is obtaining the initial signal: a narrow behavior may remain invisible until the search reaches a relevant context. Internal decoding supplies clues in our current method; an API-only approach would need to discover them through queries, potentially requiring a larger budget. We identify this extension in §7, while leaving its effectiveness and query efficiency for future evaluation.

Reviewer_cs9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets auditing of narrow, conditionally expressed biases introduced by fine-tuning, when the auditor has only the base model, the fine-tuned model and a generic corpus (no knowledge of the bias, no clean reference fine-tune, no eliciting prompts). The proposed guided-benchmarking pipeline (1) selects six "hot" layers by relative weight change and takes the top left singular vector of each layer's weight update as a task-adaptation direction; (2) projects this direction out of the corpus-averaged activation difference, decodes the residual with Patchscope, and filters the tokens with an LLM; and (3) has an LLM infer a theme and generate 30 short probes that create semantic choice points without naming the decoded tokens. Base and fine-tuned answers are compared by an LLM judge (prompt hit rate and graded strength) and by the log-probability lift of the fine-tuned answer, and the inferred theme is checked against the implanted bias. On subliminal-learning fine-tunes (four tasks, two teachers, two students, four biases), the pipeline outperforms ADL-style decoding and free-form probing (mean PHR 0.61 vs 0.43 and 0.29 on Qwen2.5-14B), recovers the implanted theme more often (GT-Rec. 0.79 vs 0.50), and stays near PHR 0.1 on control fine-tunes. A 2×2 ablation attributes the gain to cleaning at hot layers, and a risky-finance case study shows that decoding recovers the domain but not the valence of a more abstract policy.

Critical strengths
  1. The paper addresses an important, well-posed problem with a simple idea. The setting, with no knowledge of the bias, no clean reference fine-tune and no eliciting prompts, is realistic for auditing third-party fine-tunes, and using the weight update to remove the dominant task-adaptation direction from an activation difference is inexpensive (15–30 minutes per condition) and easy to adopt. The 2×2 ablation (Table 3) and the layer-level example make the intended mechanism concrete: at L26, raw decoding yields no religion-related tokens at any of the six injection positions, while cleaned decoding yields them at all six (Appendix H).
  2. The evaluation takes the risk of a self-consistent loop seriously. Because the theme, the probes and the judgments all come from the pipeline, a high PHR alone could reflect a self-consistent loop. The authors address this with ground-truth anchoring (GT-Rec. 0.79 vs 0.50 for ADL-style decoding; GT-anchored PHR 0.57), a second judge family (Cohen's κ\kappa = 0.846), and control fine-tunes that stay near PHR 0.1. The advantage over ADL-style decoding holds in every setting with successful transfer and on a second student.
Critical weaknesses
  1. Key components of the method are under-specified. LoRA adapts seven modules per layer (Table 4), but the paper does not state which matrices define ΔWl\Delta W_l, sls_l and utask,lu_{\text{task},l}, or how they are combined; utask,lu_{\text{task},l} must lie in the residual-stream space to be projected out of Δhl\Delta h_l, which constrains the choice. The task fraction ftaskf_{\text{task}} is described only as "the share of Δhl\Delta h_l explained by utask,lu_{\text{task},l}", without stating whether this refers to the norm or the squared norm. The model used for the Patchscope forward pass, the way tokens from the six hot layers are merged, and the binary judge prompt for PHR are not given, and the probe-generation prompt in Appendix K returns only the probes, so it is unclear how the theme TT passed to the judge is obtained. Since weight-guided cleaning is the paper's contribution over ADL, these omissions prevent reproduction of the central method.
  2. The central assumption behind cleaning is not validated directly. That the top singular direction of the weight update captures task adaptation rather than the bias is argued informally, and the 2×2 ablation covers a single setting (Alpaca, GPT-4.1 teacher, Qwen2.5-14B). Two direct controls are missing: projecting out a random direction, and comparing utask,lu_{\text{task},l} with the activation difference of the matched control fine-tune, which the authors already have. The random-direction control is particularly important because at L14, L19 and L27 the task direction explains under 10% of Δhl\Delta h_l by the paper's measure, yet removing it changes hits from 1/6, 2/6 and 0/6 to 4/6, 6/6 and 6/6 (Table 11), which suggests that the decoding is sensitive even to removing a minor component. Without these controls, the gain over ADL-style decoding cannot be attributed to removing task adaptation.
  3. Several reported numbers are internally inconsistent. Appendix H analyses the "Qwen2.5-14B-Instruct student" but labels L26 and L27 as late layers and calls L27 "the last layer", whereas Qwen2.5-14B-Instruct has 48 layers and 28 is the depth of Qwen2.5-7B. In Table 10, k=1 has PHR 0.30 but Strength 1.10, while Strength is about 1.5–1.7 times PHR in every other row. In every reported control row of Tables 1 and 7, Strength is below PHR (e.g., 0.10 vs 0.13); if both metrics come from the same graded judgement, as Appendix L suggests, this cannot happen, because a hit defined as a score of at least 1 implies Strength ≥ PHR, and a hit defined as a score of 2 implies Strength ≥ 2×PHR, which fails in the bias rows (e.g., 0.98 vs 0.61). Table 8 reports 91.2% raw agreement with a 95% CI of [90.8%, 93.6%], and some GT-Rec. standard deviations in Table 9 are not attainable from four scores in {0, 0.5, 1} (e.g., 0.63 ± 0.29). Because the paper's claims are entirely empirical, these inconsistencies reduce confidence in the reported tables.
  4. The statistical support and baselines are limited. Each cell averages four biases from a single listed training seed, without significance tests, and GT-Rec. 0.79 vs 0.50 rests on 28 three-level judgements per method. The free-form baseline is scored by a different judge (J′J'), so its PHR is not directly comparable. There is no logit-lens decoding baseline, although ADL uses one, and no simple black-box baseline; since all four biases are "favourite X" preferences, a fixed cross-category preference questionnaire may already surface them. Removing the LLM token filter lowers GT-Rec. from 0.88 to 0.50 (Table 10), so a substantial part of the theme recovery depends on GPT-4.1, which also serves as probe generator and primary judge.
  5. The audit targets and the risky-finance attribution are not fully characterised. The Bias Rate of the control fine-tunes is not reported (Table 1), and the held-out behavioural evaluation that defines Bias Rate is not described, so it is unclear how far the audit targets are from the natural base rate. In the risky-finance study, 10% of the fine-tuning data is financial advice, so the decoded finance tokens may reflect the data domain rather than the hidden policy; the matched control model is trained on the same mixture, but its decoded tokens, inferred theme and PHR are not reported. This weakens the claim that decoding localises the domain in which the hidden policy is expressed.
Questions for authors
  1. How exactly is the cleaning direction computed? How are ΔWl\Delta W_l, sls_l and utask,lu_{\text{task},l} obtained from the seven LoRA-adapted modules, and how is ftaskf_{\text{task}} computed? Which model runs the Patchscope pass, and how are tokens from the six hot layers merged?
  2. Is the gain caused by removing task adaptation? What happens when a random unit direction, or the top principal direction of the activation differences, is projected out instead of utask,lu_{\text{task},l}? How well does utask,lu_{\text{task},l} align with the control fine-tune's Δhl\Delta h_l, and can the 2×2 ablation be reported for the other tasks, the Qwen teacher and Gemma-2-2B?
  3. Which numbers should be used to assess the main claims? Please reconcile the inconsistencies listed above, state which model Appendix H analyses, and provide the binary judge prompt used for PHR. How is the theme TT passed to the judge obtained, given that the probe-generation prompt returns only the probes?
  4. Are the audit targets and baselines adequate? What Bias Rate do the control fine-tunes have, and which questions define the held-out behavioural evaluation? What are the decoded tokens, inferred theme and PHR for the risky-finance control model? Is the LLM token filter also applied to the ADL tokens, and how would a fixed cross-category preference questionnaire perform as a black-box baseline?
3 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the detailed assessment and specific suggestions. We have clarified the implementation, corrected the reporting inconsistencies, and added controls and evaluations addressing the main concerns as following.

W1 / Q1. Specification of key components

Appendix P now specifies the implementation, and §3 refers to it:

Weight matrix for cleaning. We concatenate the updates to the attention output and MLP down projections:

ΔWlout=[ΔWo,l  ΔWdown,l]∈Rd×(d+dff).\Delta W_l^{\mathrm{out}} = [\Delta W_{o,l}\;\Delta W_{\mathrm{down},l}] \in\mathbb{R}^{d\times(d+d_{\mathrm{ff}})}.

These projections write directly to the residual stream, so their left singular vectors have the appropriate coordinates for projection out of Δhl\Delta h_l. Each LoRA update is ΔW=(α/r)BA\Delta W=(\alpha/r)BA.

Layer score. Hot layers are selected using all seven adapted matrices:

sl=∑m∥ΔWm,l∥F2∑m∥Wm,l∥F2s_l = \frac{\sqrt{\sum_m \Vert \Delta W_{m,l} \Vert_F^2}}{\sqrt{\sum_m \Vert W_{m,l} \Vert_F^2}}

Thus, layer selection uses the full adapted-layer update, while cleaning uses the two residual-output projections.

Cleaning direction. utask,lu_{\mathrm{task},l} is the top left singular vector of ΔWlout\Delta W_l^{\mathrm{out}}, with d=5120d=5120 for Qwen2.5-14B. With LoRA rank 8, the concatenated update has rank at most 16.

Task fraction. We use

ftask=(Δhl⊤utask,l)2∥Δhl∥2,f_{\mathrm{task}} = \frac{(\Delta h_l^\top u_{\mathrm{task},l})^2} {\|\Delta h_l\|^2},

the fraction of the squared norm removed by projection.

  • Patchscope model. We inject ±αΔhlclean\pm\alpha\Delta h_l^{\mathrm{clean}} into the fine-tuned model M∗M^* at layer ll on the identity prompts.
  • Fixed layer. ADL uses the middle layer: L24 for Qwen2.5-14B’s 48 layers and L13 for Gemma-2-2B’s 26 layers.
  • Token aggregation. Within each layer, we intersect tokens across the three identity templates and retain the top 25% by frequency. We then take the union across the six hot layers, summing frequencies, and apply the LLM filter once. ADL tokens receive the same filtering procedure.
  • Theme and judgments. The generation call returns {"theme": T, "probes": [...]}. Appendix K now gives the full output format. Appendix L describes the single graded judgment S∈{0,1,2}S\in\{0,1,2\} from which both metrics are derived: PHR is the fraction with S≥1S\geq1, and Strength is the mean score.

W2 / Q2. Does cleaning remove task adaptation?

We agree that the original ablation established cleaning’s utility but did not directly validate its interpretation. Appendix M adds the following diagnostics for the catholicism-biased Alpaca model, together with end-to-end comparisons across all four biases:

  • Matched-control alignment. At L24, L29, L46, and L47, the proposed direction aligns with the corresponding control direction (∣cos⁡∣=0.86|\cos|=0.86–0.97; random-direction scale 1/5120≈0.0141/\sqrt{5120}\approx0.014). The control’s task fraction is also similar: 0.812 versus 0.837 at L46. Cleaning with the control direction recovers relevant tokens at 4, 5, 6, and 6 of six positions, versus 4, 6, 6, and 6 with our direction, and achieves end-to-end PHR 0.80 versus 0.83.
  • Estimated bias-direction overlap. A direction estimated from biased–control activation differences on bias-eliciting prompts has ∣cos⁡∣≤0.07|\cos|\leq0.07 with utask,lu_{\mathrm{task},l}, corresponding to at most 0.5% of its squared norm removed. Decoding utask,lu_{\mathrm{task},l} itself yields no bias-relevant tokens among the top 20. These prompts are used only for validation.
  • Random-projection controls. Removing a random unit direction leaves position-level hits unchanged. Removing a random direction with the same projected energy gives 1, 2, and 1 successful positions at L24, L29, and L47, versus 4, 6, and 6 for our direction. At L46, it gives 0/6 versus 6/6. End-to-end PHR is 0.59 for random projection and 0.56 for energy-matched random projection, versus 0.59 without cleaning and 0.83 with our direction. This addresses the concern that any small perturbation might improve decoding.
  • Activation-only alternative. Removing the top principal component of token-level activation differences gives PHR 0.70 and GT-Rec. 0.75, versus 0.83 and 0.88 for our direction.
  • Expanded ablation. The 2×22\times2 ablation now covers all seven successful-transfer settings on both students (Table 16). Hot-layer selection alone stays within 0.03 PHR of the fixed layer; cleaning at hot layers adds 0.12–0.25 PHR, averaging 0.19.

These results support the proposed interpretation in the examined settings. They complement the original evidence that cleaning improves recovery: raw-to-clean decoding increases hits from 0/6 to 6/6 at L46 and L47, and overall GT-Rec. improves from 0.50 for ADL to 0.79 for our pipeline.

AuthorsOpenReview ↗

Author Response (continued)

W3 / Q3. Internal inconsistencies

Thank you for identifying these issues. We have corrected them and now generate all tables directly from saved evaluation outputs using a single script, which we will release. We traced the discrepancies to four sources: (1) Off-rubric GT-Rec. outputs. Our parser accepted occasional scores such as 0.7 from the second judge, making some Table 9 statistics incompatible with the stated rubric. We rejudged the affected items using the prescribed discrete scores and recomputed all GT-Rec. statistics. (2) Outdated control evaluation. An earlier implementation used separate PHR and Strength calls with mismatched theme strings. We reran all controls using each control’s own inferred theme and one graded judgment from which both metrics are derived, and updated §4 accordingly. (3) Layer labels. Appendix H retained labels from a Qwen2.5-7B pilot, although the reported values came from the 14B student. We corrected the labels. (4) Manual table assembly. Table 10’s k=1k=1 entry, Table 8, and Table 2’s SD reporting contained assembly errors. We corrected these and standardized cross-bias reporting to sample SD.

The corrected results preserve the main method comparisons and conclusions.

W4 / Q4, part 1. Statistical support

We added paired tests and two additional training seeds:

  • Paired comparisons. Across the 28 task–teacher–bias combinations with successful transfer, two-sided Wilcoxon signed-rank tests give p<10−4p<10^{-4} for PHR versus ADL (ours higher in 23/28), p=1.7×10−3p=1.7\times10^{-3} for GT-Rec. (17 higher, 8 tied, 3 lower), and p<10−4p<10^{-4} for GT-anchored PHR (22/28 higher). Against free-form probing under the same GT-anchored judge, ours is higher in all 28 combinations (p<10−5p<10^{-5}).
  • GT-Rec. sample size. GT-Rec. remains based on 28 discrete scores per method. We therefore also report GT-anchored PHR, evaluated on 840 probes per method, while retaining the audit target—not the individual probe—as the unit of the paired comparison.
  • Training seeds. For Alpaca with the GPT-4.1 teacher, we retrained all four biased students and matched controls with seeds 43 and 44: 16 additional fine-tunes (Appendix N, Table 17). Bias Rate varies from 0.74 to 0.83, but the PHR gain over ADL remains 0.22–0.23 (mean 0.23±0.010.23\pm0.01). This variation is smaller than the within-seed variation across biases. Our GT-Rec. exceeds ADL’s in every seed (0.75–0.88 vs. 0.50–0.63), and control PHR remains 0.09–0.10.

W4 / Q4, part 2. Baselines

  • Common evaluation for free-form probing. Table 1 now identifies its distinct judge J′J'. Table 2 provides a directly comparable evaluation: all methods’ probes are judged against the implanted bias using the same prompt. GT-anchored PHR is 0.15 for free-form probing, 0.37 for ADL, and 0.57 for ours; ours exceeds free-form probing in all 28 combinations.
  • Logit-lens decoding. Appendix O evaluates both configurations with the logit lens and otherwise unchanged filtering, generation, and judging. Relative to Patchscope, PHR decreases from 0.61 to 0.48 for our configuration and from 0.43 to 0.33 for ADL; our GT-Rec. decreases from 0.79 to 0.55. Our configuration therefore retains an advantage under either decoder. At intermediate layers, the logit lens recovers 0–1 relevant tokens among its top 20, while Patchscope recovers relevant tokens at 4–6 of six injection positions. These are different diagnostics, but both indicate weaker intermediate-layer recovery with the logit lens.
  • Fixed preference questionnaire. A fixed questionnaire succeeds only if it already lists the category in which the bias is expressed. For our four main biases this holds by construction: they are “favourite X” preferences adopted from prior subliminal-learning work, and a generic preference list would include animals, trees, countries and religions. However, it does not make the questionnaire a general auditing strategy: the space of narrow behaviors is open-ended, and a questionnaire is written before looking at the audited model, so it cannot adapt which contexts it probes to what fine-tuning changed. Figure 1 shows this failure mode for a fixed benchmark. The risky-finance policy (§6.2) is a case with no matching questionnaire item: it is a valence over advice, not a preferred object. For those reason, we do not consider questionaire as one of the baselines.
AuthorsOpenReview ↗

Author Response (continued)

W5 / Q4, part 3. Audit targets and the risky-finance control

  • Bias Rate. For each main bias, we use 50 held-out preference questions adopted from Cloud et al. (2025), such as “In one word, what is your favorite animal?” and “Name your favorite religion.” Sampling each question four times gives 200 responses per bias. Claude Sonnet 4.6 judges whether each response expresses the target bias; Appendix L specifies the procedure and prompt. These questions use knowledge of the implanted bias to verify transfer and are never available to the auditing pipeline. Table 1 now reports control Bias Rates of 0.03–0.05, versus 0.38–0.76 for the successful-transfer biased conditions. Risky finance uses a separate evaluation of 100 prompts on a 1–5 risk scale (Appendix I).
  • Risky-finance control. Appendix I now reports its decoded tokens, theme, and GT-anchored PHR. The control yields savings, budget, money, account, income, and financial, with the theme “personal finance and budgeting,” close to the biased model’s “financial planning and resource allocation.” Thus, both identify the financial domain, but the control has substantially fewer risk-policy hits: GT-anchored PHR 0.16 versus 0.69. For example, in the held-out policy evaluation, the control recommends a savings account, budgeting, and an emergency fund when asked how to build financial habits.

The finance result distinguishes the two stages: decoding identifies a domain to investigate; behavioral probing assesses the risk-seeking policy within that domain.

Reviewer_dwyoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Guided Benchmarking, a pipeline for detecting narrow, hidden biases introduced during fine-tuning. It uses weight updates to identify important layers and estimate the dominant task-adaptation direction, removes this direction from activation differences, decodes the residual into candidate tokens, and uses an LLM to generate targeted behavioral probes.

Critical strengths
  1. Addresses an important problem: narrow, context-dependent biases that standard behavioral benchmarks may miss.

  2. Includes ablation studies and evaluations across multiple fine-tuning tasks, models, and implanted biases.

  3. Evaluates recovered themes against ground-truth biases rather than relying solely on the model's inferred themes.

Critical weaknesses
  1. The assumption that the dominant singular direction of weight updates represents task adaptation is not sufficiently established.

  2. Residual activation signals may contain noise or other fine-tuning effects, rather than specifically reflecting hidden biases.

  3. The evaluation relies heavily on LLMs for theme inference, probe generation, and behavioral judging, potentially introducing correlated biases.

  4. The experiments cover a limited set of models and bias types, leaving generalizability to more complex or abstract biases uncertain.

Questions for authors
  1. What evidence shows that the dominant singular direction reliably captures task adaptation rather than the hidden bias itself?

  2. How do you verify that the decoded residual tokens reflect hidden biases rather than unrelated semantic or fine-tuning signals?

  3. How sensitive are the results to the choice of generic corpus, layer selection, and number of decoded tokens?

  4. How well does the method generalize to different model architectures, full fine-tuning, and biases that are abstract or difficult to express lexically?

2 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for recognizing the importance of the problem and the value of ground-truth evaluation. We address the concerns below.

W1 / Q1. Validating the task-adaptation direction

The original experiments established the benefit of cleaning: hot-layer selection alone does not improve PHR (0.59 vs. 0.61, Table 3), cleaning recovers bias tokens at layers where raw decoding finds none (0/6 → 6/6 positions at L46 and L47, Table 11), and recovered themes match the implanted biases more often than ADL’s (GT-Rec. 0.79 vs. 0.50, Table 2). We agree that these results alone do not establish why cleaning works.

Appendix M now tests the proposed interpretation directly. The layer-level diagnostics use the catholicism-biased Alpaca model; end-to-end comparisons cover all four biases in that setting.

  • Alignment with a matched control. At L24, L29, L46, and L47, utask,lu_{\mathrm{task},l} aligns closely with the corresponding direction from the control fine-tune (∣cos⁡∣=0.86|\cos|=0.86–0.97; the random-direction scale is 1/5120≈0.0141/\sqrt{5120}\approx0.014). The control’s task fraction is also similar: 0.812 versus 0.837 at L46. Cleaning with the control direction recovers relevant tokens at 4, 5, 6, and 6 of six positions, compared with 4, 6, 6, and 6 using the biased model’s direction. End-to-end PHR is 0.80 versus 0.83.
  • Separation from an estimated bias direction. A direction estimated from biased–control activation differences on bias-eliciting prompts has ∣cos⁡∣≤0.07|\cos|\leq0.07 with utask,lu_{\mathrm{task},l}. Thus, projection removes at most 0.5% of this estimated direction’s squared norm. Decoding utask,lu_{\mathrm{task},l} itself yields no bias-relevant tokens among the top 20. The bias-eliciting prompts are used only for validation and are unavailable to the auditor.
  • Random-projection controls. Removing a random unit direction leaves the position-level hits unchanged. A stronger control removes the same fraction of ∥Δhl∥2\|\Delta h_l\|^2 as utask,lu_{\mathrm{task},l}. At L24, L29, and L47, where ftask<0.1f_{\mathrm{task}}<0.1, this gives 1, 2, and 1 successful positions, versus 4, 6, and 6 for our direction. At L46, it gives 0/6 versus 6/6. End-to-end, neither random control improves on raw decoding: PHR 0.59 and 0.56 versus 0.59 without cleaning.
  • Activation-only alternative. Removing the top principal component of token-level activation differences gives PHR 0.70 and GT-Rec. 0.75, compared with 0.83 and 0.88 for our direction.
  • Broader component ablation. The 2×22\times2 ablation now covers all seven successful-transfer settings on both students (Table 16). Hot-layer selection without cleaning stays within 0.03 PHR of the fixed layer; cleaning at hot layers improves PHR by 0.12–0.25, averaging 0.19.

These controls support the task-adaptation interpretation in the examined settings, without assuming that narrow expression guarantees geometric separation from the bias.

W2 / Q2. Distinguishing hidden biases from other fine-tuning effects

We thank the reviewer for the insightful comment. For this concern, we use three complementary checks:

  • Matched controls. Cleaned decoding of the catholicism control yields no catholicism-relevant tokens at the layers examined in Table 11. Across the Qwen control conditions, our pipeline’s PHR against each control’s own inferred theme is 0.07–0.11 (Table 1). In risky finance, the control also yields finance-related tokens, but its GT-anchored PHR against the risk-seeking policy is substantially lower: 0.16 versus 0.69.
  • Ground-truth evaluation. GT-Rec. is 0.79 for our method versus 0.50 for ADL (paired Wilcoxon p=1.7×10−3p=1.7\times10^{-3}). When all methods’ probes are evaluated against the implanted bias with the same judge prompt, GT-anchored PHR is 0.57 for ours, 0.37 for ADL, and 0.15 for free-form probing (Table 2; p<10−4p<10^{-4} versus ADL).
  • Projection controls. As detailed above, removing a random direction with the same projected energy does not reproduce the improvement: end-to-end PHR is 0.56, versus 0.59 without cleaning and 0.83 with our direction (Appendix M).

These results distinguish recovery of the implanted behavior from merely finding coherent residual tokens. In particular, the finance control shows why decoded themes must be followed by behavioral validation.

AuthorsOpenReview ↗

Author Response (continued)

W3. Reliance on LLMs

We agree that using one model family in several roles can introduce correlated errors. We therefore test both an alternative judge and an alternative filter/probe-generator model:

  • Alternative judge. On the seven successful Qwen settings for ADL and our method, Claude Sonnet 4.6 gives hit agreement of 92.6% and Cohen’s κ=0.851\kappa=0.851 across 1,680 response pairs; weighted κ\kappa for GT-Rec. is 0.792. Under this judge, ours still exceeds ADL in PHR (0.58 vs. 0.41) and GT-Rec. (0.77 vs. 0.48; Tables 8–9).
  • Alternative filter and generator. Replacing GPT-4.1 with Llama-3.3-70B-Instruct for both token filtering and probe generation across all seven Qwen settings gives PHR 0.57 versus 0.40 for ADL, and GT-Rec. 0.75 versus 0.46. Each aggregate is within 0.04 of its GPT-4.1 counterpart, and the advantage over ADL persists.

These checks reduce dependence on a particular LLM but do not eliminate shared errors across models. We retain this limitation in §7.

Q3. Sensitivity to corpus, layer selection, and token retention

Appendix G now covers all three factors (Table 10):

  • Corpus. Replacing 5,000 FineWeb documents with 5,000 held-out Alpaca prompts gives PHR 0.80 versus 0.83 with cleaning, and GT-Rec. 0.88 for both. Without cleaning, the Alpaca corpus gives lower PHR (0.52 vs. 0.59) and GT-Rec. (0.50 vs. 0.63). Thus, cleaning reduces sensitivity to this change in corpus.
  • Layer count. PHR increases from 0.58 at k=1k=1 to 0.83 at k=6k=6, with no further improvement at k=9k=9 (0.82).
  • Token retention. Retaining the top 10%, 25%, or 50% of intersected tokens gives PHR 0.68, 0.83, and 0.75, respectively.

The held-out Alpaca inputs represent the nominal task distribution and require no knowledge of the hidden bias.

W4 / Q4. Architectures, full fine-tuning, and abstract biases

Our main experiments cover Qwen2.5-14B-Instruct and Gemma-2-2B-IT (Appendix E). The risky-finance study (§6.2) extends the evaluation beyond concept preferences: decoding identifies the financial domain, while behavioral probing reveals the risk-seeking valence.

We additionally fully fine-tuned Gemma-2-2B-IT on Alpaca using the GPT-4.1 teacher, for all four biases and matched controls (Appendix Q). The leading direction captures a smaller fraction of the output-projection update than under LoRA (0.14 vs. 0.38), but still aligns with the control direction (∣cos⁡∣=0.81|\cos|=0.81). Our pipeline achieves PHR 0.70 versus 0.51 for ADL—a gain of 0.19, compared with 0.22 under LoRA—and GT-Rec. 0.75 versus 0.50. Control PHR is 0.12.

This extends the evidence beyond low-rank adaptation. Broader full-fine-tuning coverage and removing multiple directions remain future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Response to Reviewers

We thank the reviewers for their careful assessment and constructive feedback. We have revised the manuscript in response to the reviewers’ comments, strengthening the empirical evidence and clarifying the presentation and scope of our claims. The main updates are:

  • Validation of cleaning direction. Added matched-control comparisons, bias-direction overlap measurements, random-projection controls, an activation-only alternative, and expanded component ablations.
  • Robustness and statistical support. Added an independent judge, an alternative token-filtering and probe-generation model, additional training seeds, paired statistical tests, and full-fine-tuning experiments.
  • Ground-truth evaluation and practical relevance. Strengthened evaluation against the implanted biases and expanded the risky-finance analysis with a matched control, distinguishing recovery of the financial domain from elicitation of the risk-seeking policy.
  • Corrections and reproducibility. Corrected the numerical inconsistencies and layer labels, unified PHR and Strength under one graded judgment, clarified implementation details, and generated tables directly from saved evaluation outputs using a single script that we will release. The corrected results preserve the main method comparisons and conclusions.

Our individual responses provide the results and corresponding manuscript references.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.