acceptodds
Under review as a conference paper at ICLR 2027

Steer, Then Classify: Adaptive Document-specific Representation Steering for Text Classification

Abstract

Text classification is a fundamental natural language processing (NLP) problem, spanning many tasks such as intent and topic classification, toxicity and stance detection, entailment, factuality assessment and many more. Despite substantial advances in large language models (LLMs), modern text classifiers still largely follow the same paradigm, that encode the input with a pretrained language model and train a classification head on top of the resulting representation. This leaves the internal representations of the model largely passive during inference. In this paper, we introduce a different and novel approach to text classification based on representation steering. Rather than directly classifying a fixed document representation, our method dynamically modifies the representation of each input document individually along learned steering directions to make task-relevant information more separable in representation space. Building on this idea, we propose AdaSteer, an adaptive steering framework that determines how and where to intervene in an LLM's representations for each input. Across the Taarof and Multilingual Toxicity Benchmark datasets, AdaSteer consistently improves classification performance over conventional classification heads and existing steering baselines. Notably, when both approaches are initialized from the same underlying LLM, AdaSteer achieves a 1.89% average improvement in F1 score over the standard classifier baseline across four datasets (i.e. on the Irony, Taarof, Multilingual Toxicity and Sentiment datasets), while requiring comparable trainable parameters. Our experiments further show that per-document adaptive steering produces more discriminative representations and that the gains persist across model scales and datasets, providing a complementary alternative to conventional discriminative fine-tuning, and establishing representation steering as a promising paradigm for text classification. Our entire codebase will be released for facilitating reproducibility.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Text Classification, Dynamic Document-specific Representations, Activation Steering, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f4Q1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a simple intervention for LLM-as-classifier. The work builds on CAA (Rimsky et al.), which constructs a task-specific steering vector by computing the mean difference between hidden-state representations associated with two classes. The main extension proposed in this paper is the introduction of an instance-specific steering strength, which is combined with the task-specific steering vector during inference. Experiments are conducted on six binary classification datasets, and the authors report that the proposed AdaSteer method outperforms baseline (no steering). However, I did not find the description of the baseline and had difficulty understanding exactly how they (both baseline and proposed method) are implemented.

Critical strengths
  • The high-level idea of the proposed method makes sense and is easy to understand.
  • The authors conduct statistical significance tests to determine whether the observed improvements over the baseline are statistically significant.
Critical weaknesses
  • The writing quality is not great, and I found many of the technical details difficult to follow. In particular, Section 3.2, which describes the core proposed method, is not well written. I still could not fully understand it after several readings, and I believe the description lacks sufficient detail.
    • Line 202: Do you mean eight random numbers sampled separately for each example, or are they fixed for all examples?
    • Lines 205-206: What does zz mean?
    • Line 220: Why are the values +3 and -3 used here? Are they related to the range [−1,1][-1, 1] defined in Line 202?
    • Line 232: predict the document-specific steering strength using a lightweight Multilayer Perceptron is confusing. My understanding is that the method evaluates several candidate steering strengths, obtains classifier logits, and uses them to construct a document representation. However, it is not clear how the steering strength itself is predicted.
    • Line 256: The text states that the α\alpha predictor produces a magnitude, whereas the earlier description suggests that α\alpha is a scalar?
    • I also fail to understand the claim that, with a frozen LLM, the proposed method modifies the model's internal computation What do these internal computation refer to, and how are they being altered?
  • There is very little discussion of the experimental results. Section 4.4 mainly describes what was done and references tables and figures, rather than describing the findings. The authors should provide a clearer summary of the key results. In addition, the claims made in Section 5 should be more explicitly linked to the empirical evidence. Rather than repeatedly saying as shown in Tables 1, 2, 3, 4, and 5 the authors should explain which specific results support each claim and why.
Questions for authors
  • All evaluated datasets are binary classification tasks. It would be interesting to investigate whether the proposed approach can be extended to multi-class and even multi-label classification settings, if technically feasible.
  • The evaluation covers four model families (Llama 3.2, Gemma 2, GPT-2, and Qwen 2.5), but these models are already somewhat outdated. It would be interesting to see whether the findings generalize to more recent LLMs.
  • Line 269: The equation is difficult to follow, and I am not sure what it is intended to represent.
  • Figure 3: I did not find this figure informative. The authors may want to consider a different visualization.
Reviewer_fxkoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AdaSteer, a document-specific adaptive representation steering framework for text classification. It employs Concept Activation Addition (CAA) to derive steering directions, with a lightweight MLP predictor estimates a steering intensity, α, for each document based on the classifier’s logits. The primary contributions are threefold: (1) it introduces representation steering as a mechanism for supervised text classification; (2) it enables the learning of input-dependent steering intensities during inference without requiring ground-truth labels; (3) it provides extensive validation across six datasets and 20 large language models (LLMs) from four different model families. The results demonstrate that this approach consistently outperforms traditional classification heads and existing steering baselines, yielding significant F1-score improvement over standard classifiers. These findings are further supported by evaluations involving LoRA and non-LoRA configurations, ablation studies on CAA sample sizes, latency testing, and interpretability analysis via activation patching.

Critical strengths

One of the greatest strength of this paper lies in the breadth and diversity of the evaluation: the six datasets span phenomena deeply rooted in cultural pragmatics—such as Taarof, politeness, sentiment, and sarcasm—as well as general-purpose tasks like multilingual toxicity detection. This goes far beyond the typical single-task validation, allowing the robustness of the proposed method to be tested across genuinely distinct linguistic phenomena. This echoes with the point that the benefits of "document-specific prompting" should logically vary depending on the nature of the task. The paper’s own analysis confirms this—showing greater gains for sarcasm and Taarof compared to sentiment and toxicity—which demonstrates that the dataset design directly substantiates the core claims rather than serving merely as an ornamental addition. Furthermore, by incorporating 20 large language models (LLMs) from four different model families—including both base and instruction-tuned variants—the study ensures that the reported performance gains can be considered a general effect, rather than an incidental result specific to a single model.

Critical weaknesses

One of the most critical weakness is the absence of a proper baseline: the paper never compares its approach against CAA with a fixed guidance strength, nor against any existing adaptive guidance methods. Consequently, the core claim—that "document-specific α is the source of the gains"—is never truly isolated; readers cannot discern whether the improvements stem from the guidance mechanism itself or from the underlying predictor. This issue is compounded by significant performance regressions observed in several Qwen and GPT configurations —results that the paper reports without sufficient explanation or mitigation strategies.

Furthermore, layering a learned scalar onto the established CAA framework is incremental and largely empirical —and lacks comparisons against strong, GPT-class baselines. As a result, its practical significance relative to current state-of-the-art practices remains unestablished.

Questions for authors

The paper states that the complete codebase will be released. Please specify where it will be made available (repository link/anonymous link), including but not limited to CAA-guided directions extracted by each model and checkpoints of the trained α-predictor and their corresponding data splits.

Reviewer_ynJhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaSteer, a supervised text-classification framework that adaptively intervenes on a decoder language model’s internal representations before classification. AdaSteer first constructs a task-specific steering vector using the mean representation difference between length-matched examples from the two classes. For each training document, the authors evaluate eight steering coefficients in −1,1-1,1 and use the probability assigned to the ground-truth class to construct a soft target coefficient. A lightweight MLP then learns to predict this document-specific coefficient from the classifier logits obtained under three probe interventions. At inference, the predicted coefficient is applied to the steering vector before the final classification decision. The method is evaluated on six English binary-classification datasets using 20 decoder models from the GPT-2, Gemma 2, Llama 3.2, and Qwen 2.5 families, in both frozen and LoRA settings. The experiments report classification metrics, significance tests, sensitivity to the number of examples used for steering-vector construction, latency, throughput, and activation-patching results. The authors conclude that document-specific steering generally improves classification and provides a complementary alternative to conventional classification heads.

Critical strengths
  1. Relevant problem and clear motivation. The paper studies whether changing a document's representation before classification can improve performance. It is reasonable that easy and difficult documents may need different steering strengths.

  2. Simple and interpretable method. AdaSteer uses a shared steering direction vv and predicts a different steering strength α^\hat{\alpha} for each document, with h′=h+α^vh' = h + \hat{\alpha}v. The predictor does not require the true label during inference and adds few trainable parameters.

  3. Large-scale experiments. The paper evaluates six datasets and 20 models from four model families, with both frozen and LoRA settings. The comparisons use matched unsteered baselines, some settings show strong gains, and the paper studies how the amount of data used to build the steering direction affects performance.

  4. Efficiency is considered. The paper reports latency and throughput, which is important because AdaSteer requires several model forward passes.

  5. Several forms of analysis are included. The paper reports accuracy, F1, ROC-AUC, statistical tests, and activation-patching results. These analyses aim to measure both performance and changes in the model's internal computation.

Critical weaknesses

The first three weaknesses are the most important to my assessment. The remaining points would strengthen the paper but are not, by themselves, decisive.

  1. The results do not support the claim of consistent improvement. Across the reported settings, roughly half improve and roughly half worsen. Several settings show large performance drops. The paper also does not explain how the claimed 1.89% average F1 improvement was calculated. It is therefore difficult to reconcile this claim with the reported tables.

  2. The paper provides little analysis of its failures. The paper does not study why AdaSteer causes large performance drops in some settings. Useful analyses would include the distribution of predicted steering strengths, how often the predictor chooses the wrong direction, whether its outputs reach the allowed limits, and how steering strength relates to baseline confidence and document difficulty. Without this analysis, it is unclear when AdaSteer can be used reliably.

  3. The benefit of the steering-strength predictor is not isolated. The paper does not compare AdaSteer with a fixed-strength CAA baseline whose steering strength is selected on validation data. As a result, it is unclear whether the gains come from adaptive steering or from steering itself. A fixed-strength CAA baseline, an oracle steering strength, and a classifier that directly uses the probe logits would help answer this question.

  4. The efficiency trade-off is not fully reported. AdaSteer appears to require three probe forward passes and one final forward pass, compared with one forward pass for the unsteered baseline. Reporting latency is a strength, but the paper does not summarize the average latency or FLOP multiplier, explain whether intermediate computation is reused, or compare with a method given the same compute budget. I view this as an important practical concern rather than a decisive weakness.

  5. The ROC-AUC and calibration evaluation is incomplete. The ROC curves appear to include only AdaSteer, without matched results for the unsteered baseline. The paper also does not report how the decision threshold is chosen or whether steering changes probability calibration. These results would help determine whether lower accuracy and F1 come from worse class ranking or from shifted prediction scores. I view this as a request for further analysis rather than a central weakness.

Questions for authors

The main questions are already described in the weaknesses above. In particular, I would like the authors to explain the 1.89% result, analyze the large performance drops, compare with fixed-strength CAA, and clarify the performance-cost trade-off. Answers to these questions could change my assessment.

I also have the following clarification questions:

  1. Representation used for classification: Section 4.3 states that the classifier uses the final non-padding-token representation, while Appendix A.2.1 states that it uses masked mean pooling. Which representation was used in the reported experiments? Was the same representation used to build the steering direction?

  2. Hyperparameter choices: How were the probe steering strengths {−3,0,+3}\{-3,0,+3\}, the target range [−1,1][-1,1], layer L−2L-2, temperature τ=5\tau=5, eight-point sweep, and 1,000 predictor-training epochs selected? More explanation or ablations, including in the appendix, would be helpful.

  3. Reproducibility details: Please provide the train, validation, and test splits; the number of random seeds; the training duration; the early-stopping procedure; and the method used to select hyperparameters and decision thresholds. These details would improve reproducibility, although they are not the main reason for my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.