acceptodds
Under review as a conference paper at ICLR 2027

Learning to Decide, Not to Reason: Parameter-Efficient Decision Operators via Low-Rank Activation Steering

Abstract

Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline. We introduce , a System-1 decision operator trained by behavior cloning that lowers this cost by roughly two orders of magnitude. The default operator uses 330K parameters to match a 1.33M-parameter operator trained with reinforcement learning, exceeds or achieve comparable performance, while collapsing 3,685-token deliberation into a 6-token decision with no loss in accuracy. A rank-4 variant with 23K parameters, 1/58 of the strongest published skill operator, suffices for SearchQA and near-suffices for LiveMath, where higher rank still helps; the same recipe transfers across five tasks and three backbones, with out-of-distribution gains persisting on LiveMath problems released months after training. The gap to prior work is trainability, and it is set jointly by initialization and architecture: the initialization of prior operators zeroes the gradient of both large factor matrices at the first optimization step, whereas our zero-initialized output projection inside a shared low-rank backbone receives a gradient immediately, which a gradient-flow probe confirms directly. The gain isn't chain-of-thought compression: 23 of 57 LiveMath points beat the base model's best-of-8 sampling, and a logit-lens probe shows the operator amplifies the answer along the model's existing late-layer pathway, not writing it earlier. Gains track the base model's headroom across 13 base–task pairs, and skills compose as approximately linear operators that can be added, interpolated, and hot-swapped at inference time. Code on https://anonymous.4open.science/r/decisionsteer-DBE4/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
activation steering, parameter-efficient adaptation, system 1 thinking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K8PkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DecSteer, a lightweight residual operator trained with behavior cloning. It reports performance comparable to the RL-trained KV-Skill at substantially lower parameter and inference cost.

Critical strengths

The reported improvements in parameter efficiency and inference costs are of practical value, and the paper includes a number of diagnostic analyses.

Critical weaknesses
  1. First, supervised learning for parameter-efficient adaptation and skill compression has already been well researched (e.g., ReFT and SoftSkill). The original KV-Skill paper itself reports that BC significantly outperforms RL on ALFWorld based on successful trajectories (69.4 vs. 42.3), so the fact that BC can work as a residual operator is not new (I am referring to the original Version 1 of KV-Skill, which was published prior to this submission). Therefore, a central claimed contribution is the explanation that the V=0 anchoring prevented training with BC. However, this causal explanation is not supported by KV-Skill’s own BC control experiments. V=0 operators can be trained well under BC on DocVQA using both the same 4B backbone and a larger backbone. LoRA, on the other hand, which does not use V=0 anchoring, fails under BC on exactly the same 4B tasks where KV-Skill also fails. The authors’ own new implementation (Table 4) also trains successfully with V=0. These results suggest that failure depends more on the task, the backbone, or the training recipe than on initialization alone, thereby providing insufficient support for the central mechanistic explanation.

  2. All five benchmarks are QA-style tasks with explicit answer supervision. KV-Skill, by contrast, also evaluates many interactive and long-horizon settings (such as ALFWorld). A purely response-based BC is not directly applicable when no reference answers or expert demonstrations are available—and this is precisely the scenario that motivates RL. Nevertheless, the paper concludes that “reinforcement learning is unnecessary” and presents its operator as a general System 1 decision-making mechanism. Also, the paper does not clarify whether its efficiency advantage persists beyond answer-supervised tasks or whether the simplification sacrifices capabilities supported by the original design. These broader claims are therefore not justified by the evidence presented.

Questions for authors
  1. Since KV-Skill’s own BC control experiments show that LoRA fails on the same 4B tasks, while V=0 operators succeed on other tasks and backbones, what evidence suggests that the failure of BC is due more to the initialization than to the task, the backbone, or the recipe? Could the authors conduct an “initialization × architecture” ablation analysis for Qwen3.5-4B using an identical recipe?

  2. Can DecSteer learn a skill from reward alone, without gold answers or demonstrations, on the long-horizon tasks evaluated in KV-Skill (e.g., ALFWorld, OfficeQA)? If not, how should the claim that RL is unnecessary be qualified?

  3. Under matched supervision and parameter budgets, how does DecSteer compare with ReFT and with a simple linear readout on the base model's late-layer states, given that the logit-lens analysis indicates the answer is already present there?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response: We Would Greatly Appreciate Your Feedback

Dear Reviewer K8Pk,

Thank you again for your careful review and constructive suggestions, which has been very helpful in strengthening our work.

We have posted our responses, including the clarification on the contribution and additional experiments including more ablation studies about initialization × architecture, the effect of initialization on the training (both BC and RL), extra long-horizon tasks , RL with our methods and extra baselines (ReFT and with a simple linear readout).

When you have a chance, could you please let us know whether our responses adequately address your concerns? If any questions or concerns remain, we would be more than happy to address them. We would be happy to clarify any remaining points.

Reviewer_SNMwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DecSteer, a parameter-efficient decision operator that enables frozen language models to produce accurate, concise answers without lengthy reasoning traces. The method introduces a shared low-rank activation-steering module with RMS normalization, input-dependent gating, and a zero-initialized output projection. Unlike prior reinforcement-learning-based skill operators, DecSteer is trained using behavior cloning on answer tokens only. Experiments across five tasks and three language model backbones show that DecSteer achieves performance comparable to or better than existing skill operators while using substantially fewer trainable parameters and reducing inference costs. The paper further investigates the effects of initialization on gradient flow, the required operator rank, out-of-distribution generalization, and the composability of learned skills.

Critical strengths
  1. DecSteer achieves competitive performance with substantially fewer trainable parameters than KV-Skill. The reported reduction in output tokens and inference latency is particularly impressive.
  2. The comparison between different identity-initialization strategies, supported by gradient-flow experiments, provides useful insights into why behavior cloning succeeds with the proposed architecture.
  3. The method is evaluated on five tasks and three backbone models, demonstrating substantial improvements on LiveMath and SearchQA while maintaining competitive performance on other benchmarks.
Critical weaknesses
  1. DecSteer differs from KV-Skill in initialization, architecture, normalization, and parameter sharing. More controlled ablations are needed to identify which components primarily drive the improvements.
  2. Exceeding the base model's best-of-8 sampling performance does not establish that the correct answers were inaccessible to the original model. The evidence supports improved answer extraction but not necessarily the acquisition of new reasoning capabilities.
  3. The experiments mainly involve tasks with short, directly verifiable answers. It remains unclear whether the approach extends to problems requiring substantial multi-step reasoning or open-ended generation.
  4. The substantial improvements obtained from only 35 LiveMath training examples warrant further investigation into robustness across different training subsets and label distributions.
Questions for authors
  1. Could the authors provide controlled ablations that isolate the effects of zero-initialized output projection, RMS normalization, depth sharing, and input-dependent gating? In particular, how much improvement can be attributed specifically to the initialization strategy?
  2. The paper argues that DecSteer provides gains beyond what the base model can achieve through sampling. Since best-of-8 is a finite-sampling estimate, how can the authors distinguish genuinely new capabilities from improved access to low-probability correct answers already present in the base model's distribution?
Reviewer_irtfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces DecSteer for System-1 skills trained via behavioral cloning rather than previous RL-based approaches. They note that prior work has unsuccessfully attempted using BC before, but by improving on the identity anchor, they are able to make BC an effective and cheaper alternative. They additionally analyze why DecSteer works (when prior approaches did not) and motivate individual engineering decisions, in particular, that DecSteer amplifies existing answer directions rather than surfacing answers earlier, and provides gains beyond mere compression.

Critical strengths
  • Significantly cheaper training than required for previous approaches for System-1 skills that condense deliberation into direct answer to avoid wasted inference compute.

  • Analysis of why and when their method works. This motivates engineering decisions involved in their method.

  • Empirical evidence that their skills can be combined without dramatic loss.

Critical weaknesses
  • The paper is occasionally difficult to follow and lacks context for its claims. In particular, it buries its primary finding, that System-1 skills can be trained more cheaply with BC than current RL approaches, unlike previously thought.

  • There are three components to DecSteer that feel insufficiently separated in presentation both in analysis and comparison to prior work: System-1 decisions, BC, and identity anchoring method. The motivation behind each of these choices in comparison to prior work should be clarified to highlight the novelty of this paper.

Questions for authors
  • Is it possible to combine your method for identity anchoring with RL for additional improvements?

  • If task difficulty is unclear or borderline and may require reasoning, would DecSteer hinder performance? Can DecSteer’s usefulness for a task be easily estimated?

2 comments
Reviewer_irtfOpenReview ↗

Reply to Authors

Thank you for the additional information, which have addressed most of my concerns. I would appreciate if the authors better connect their design decisions to prior work in their revisions. I retain my score.

AuthorsOpenReview ↗

Reply to Reviewer irtf

Thanks for the reply and the valuable suggestions. We have revised the paper as suggested. We added the table to show the connection between ours and the prior works. We also included the added experiment results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.