acceptodds
Under review as a conference paper at ICLR 2027

Finding the Neurons That Ask: Relation-Selective Privacy Control in Fine-Tuned LLMs

Abstract

A language model fine-tuned to extract personally identifiable information answers a request for any relation it was trained on, including relations a deployment must withhold. However, MLP neurons are polysemantic, so imprecise localization damages non-target relations. This work proposes Question-Contrast Average Precision Scoring (QCAPS), which targets the neurons that route a request rather than the knowledge it retrieves. Each neuron is scored by how well its activation separates the target question from contrastive questions over the same context. Each selected neuron is then driven below its off-state by a multiple of its own activation gap, with the multiple calibrated jointly with the selection threshold. Across five relation types on two fine-tuned extraction models, the largest overlap between any two relations' neuron sets is 71 to 250 times lower under QCAPS than under answer-presence identification. The target is suppressed to between 0.6% and 19.2% of baseline, under 10% on eight of the ten model and relation pairs, at lower cost to the remaining relations than parameter-level unlearning on nine of the ten. At a matched neuron budget, substitutions drawn from prior work leave the target at or near baseline. The intervention acts at inference time and is reversed by detaching a hook, without monosemantic decomposition or retraining.

Primary area:
interpretability and explainable AI
Keywords:
Privacy neurons, Relation-specific PII suppression, Inference-time access control, Mechanistic interpretability, Activation intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cNizOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of preventing the extraction of a target attribute in fine-tuned LLMs with relation-selective suppression of information extraction. The main contributions are three parts:

  1. QCAPS which tries to scoring neurons by how well their activations distinguish target relation questins from other questions about the same document;
  2. Neuron Forcer which tries to overwrite selected activations using a displacement proportional to their target versus contrast activation gap;
  3. Joint calibration, which tries to optimize neuron selection thresholds and intervention strength to balance target suppression against the collateral damage.

The validation mainly done with two fine-tuned models: llama-3.8B and Qwen2.5-7b, on five relations from one synthetic corpus. The results support selective suppression in this controlled setting, but effectiveness varies substantially by relation. They do not establish robust privacy protection against partial disclosure, compositional reconstruction, or adversarial bypass, nor conclusively demonstrate that the identified neurons specifically implement request routing.

Critical strengths
  1. The contrastive localization strategy is well-motivated. Holding the document fixed while varying the requested relation provides a reasonable way to distinguish question dependent activations from responses to document content. Combined with reversible inference-time intervention, this offers a useful approach to controlling extraction without retraining.
  2. The experiments demonstrate selective suppression for several model-relation pairs in the evaluated extraction setting. Across two models and five relations, the method reduces target extraction below 10% of baseline in eight of ten configurations, although sometimes with negligible collateral damage. Component substitutions and per-relation damage analyses help establish when the proposed combination works and expose its limitations. These results support the practical value of the approach, even though broader privacy robustness remains unproven.
Critical weaknesses

The evaluation does not establish robust privacy protection. It measures exact-match extraction on one synthetic corpus and tests natural paraphrases, but not adversarial questions, concept decomposition, or multi-turn reconstruction. A model could fail to produce the complete target answer while still revealing enough information to recover it. This limits the evidence for the paper’s proposed use as privacy access control.

The comparisons do not clearly establish an advantage over existing methods. Several baselines substitute individual components rather than reproduce the full methods. The LDEDE comparison replaces its original relevance measure with an activation-gap proxy. Moreover, Adaptive RMU generally achieves stronger suppression, so comparing collateral damage at different suppression levels is inconclusive. Comparisons at matched target leakage, with comparable tuning budgets, are needed to establish the claimed benefit.

Questions for authors

Clarification: Does the intervention affect only question tokens, or all tokens after the question marker, including instructions and generated answers? Equation (2) appears to specify the latter. This distinction affects whether suppression can be attributed specifically to question processing.

Clarification: Are the neuron-identification, activation-statistics, calibration, and final evaluation datasets disjoint? Appendix E.2 mentions re-estimating statistics on the evaluation split. Which reported results use these statistics?

Evidence needed for assessment: Does QCAPS retain its collateral-damage advantage when methods are compared at matched target leakage with comparable tuning budgets? Which comparisons reproduce complete baseline methods rather than component substitutions?

Evidence needed for assessment: Have you checked whether suppressed information remains recoverable through partial answers, decomposed questions, or multiple queries? If such evidence is unavailable, would you narrow the privacy claim to suppression of direct extraction under the evaluated prompts?

Reviewer_BuaXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors focus on finding LLM neurons that respond to queries (routing neurons) instead of to the documents (presence neurons). The idea seems interesting, however this paper is really hard to follow. The problem formulaiton is blurred. No overall framework is given. The experiment settings such as metric needs more explanations. The above points make it hard to value the core idea and contribution of this paper. In general, I recommend a weak reject.

Critical strengths

S1. A training-free localization strategy, Question-Contrast Average Precision Scoring (QCAPS), is proposed, which scores each neuron by how well it separates a request for the target relation from requests for the others over one document.

S2. A neuron suppression strategy driven by the target versus non-target activation gap is proposed, which displaces each selected neuron below its off-state at question-token positions.

Critical weaknesses

This paper is really hard to follow.

W1. The logic behind some statement is confusing.

In the first contribution: ''The selected population is specific to a single relation, so one relation can be addressed without disturbing how the document is represented.''

For me, it is hard to understand the logic behind.

W2. Some content in this paper is not smooth.

In 3,2, between equation (6) and (7), it says: ''It also assigns a presence-sensitive neuron, which takes similar values under all five questions, a score at the positive class rate regardless of how large its activations are.''

I wonder where the term five questions come from. Up to section 3.2, there is nothing about five questions. It just appears suddenly.

W3. The problem formulation is blurred.

In the end, it says: This work solves this problem in three stages. First, it proposes QCAPS ...

What is ''this problem''?

W4. No overall framework is given.

W5. The experiment settings such as metric needs more explanations.

Questions for authors

W1-W2

Reviewer_cxGrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Question-Contrast Average Precision Scoring (QCAPS), a method for identifying and suppressing MLP neurons associated with requests for particular PII relations, such as phone numbers, email addresses, or home addresses. Unlike approaches that identify neurons based on whether sensitive information appears in a document, QCAPS holds the document fixed and varies the requested relation. It uses average precision to identify relation-selective neurons, then applies an inference-time activation intervention that pushes these neurons below their typical non-target activation values. The selection threshold and intervention strength are calibrated jointly to balance target suppression and preservation of other relations. Experiments on fine-tuned LLaMA-3 8B and Qwen2.5-7B models demonstrate substantial reductions in exact-match extraction of targeted relations.

Critical strengths
  • The problem formulation is interesting and it has a meaningful distinction from the previous methods.
  • Provides useful component-level experiments
  • Promising empirical results showing that both neuron selection and intervention strength are important for effective suppression.
Critical weaknesses
  • What if the method gets a question like "Extract all personal information mentioned in this document." or a combination of multiple relations. Paper doesn’t provide evidence for this.
  • The motivation for replacing answer-presence-based localization with question-based localization is not fully established for broader privacy protection. While QCAPS achieves stronger selective suppression on the five predefined extraction tasks, the intervention is applied to all subsequent questions, and its effectiveness on unseen question types or indirect requests is not evaluated.
  • The experimental setting uses structured prompts requesting one of five predefined relations. Potentially it is straightforward to enforce access restrictions through request-level filtering or output validation. The paper does not compare against such simpler approaches.
  • The system assumes an overly constrained and structured set of questions. Does the method hold up against an adversary?
  • Generalisability is limited with one dataset and two models.
Questions for authors
  • Is modifying thousands of internal neurons is necessary or preferable for relation-level privacy control compared to the simpler approaches?
  • Though the authors compare against some paraphrasing, do these phrases sufficiently cover the real world possible phrasings? How similar is this paraphrasing or how different are these paraphrases?
  • Could the authors provide comparisons with the complete DEPN, APNEAP, and LDEDE methods, including their original neuron identification and intervention procedures, under the same experimental settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.