- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose training language models with a separate control channel for SAE-derived semantic features. During pretraining, features from the target segment teach the decoder to use that channel; a prediction head supplies features at inference time. The Mancelle models are evaluated for concept steering and resistance to prompt and latent counter-steering. The 400M experiments include controls matched by parameter count and by decoder architecture. The results show a promising control mechanism, alongside a substantial reduction in general capability.
The training signal is richer than a single good/bad label: it comprises positive and negative units derived from a curated SAE vocabulary. The decoder reads these through cross-attention, and a separate head learns the next segment’s features so generation can proceed without future-derived annotations. The natural-language interface maps a description into the control vocabulary. The main evaluation distinguishes concept adherence, instruction following and fluency, and then measures what remains under prompt and latent counter-pressure. This is an interesting proof of concept for a learned control interface, rather than a direct demonstration of reliable behavior in arbitrary downstream applications.
Critical strengths
-
I find the central design insightful and creative. Conditional pretraining has precedents, which the paper acknowledges, but the combination of a rich SAE-derived vocabulary, a separate encoder channel, and a training objective that rewards using the signal is a coherent contribution. The splicing procedure is intended to make the continuation harder to predict from context alone, strengthening the incentive to read the control channel. This connects the data construction, objective and architecture in a useful way. The information decomposition clarifies that incentive; its value here is the design insight rather than a new entropy identity. The paper also recognizes the difficult inference step: predicting the conditioning signal when the future text is unavailable.
The choice to make the channel valuable to prediction is the most compelling part of the paper for me. It gives the decoder a reason to learn the interface, and the feature vocabulary aims to make that interface useful for more than a few predetermined labels. This is a more interesting contribution than simply adding another auxiliary tag to the token stream. The combination also exposes a meaningful research question: how much control can be obtained while the model remains able to predict the signals it needs for ordinary generation? I would encourage the authors to emphasize this mechanism and the question it opens.
-
The small-model controls are helpful. Matching total parameters and matching the decoder answer different questions, and the paper includes both. I also appreciate the reporting of absolute scores, fluency, and capability loss alongside steering retention. Normalizing the latent perturbation by residual norm makes that comparison more interpretable.
The two 400M comparisons deserve explicit credit because neither matching choice is perfect on its own. Matching total parameters changes the decoder resources; matching the decoder leaves the conditioned model with additional components. Showing both makes it harder to attribute the entire result to one favorable choice. The full steering curves and the reporting of absolute scores also help avoid treating high retention from a weak initial steering level as a decisive success. These are thoughtful evaluation choices that strengthen the case for the idea.
Critical weaknesses
-
My main reservation is what “robust control” means here. The experiments test concept steering against a suffix search with up to eight proposals and two fitted latent directions. That is useful evidence, but it is a fairly limited attack setting. Preventing writes into the encoder does not establish that the decoder’s behavior, or its context-dependent feature predictions, cannot be redirected. An LM-judged concept score is also a different target from factual correctness or reliable tool use. The judge’s agreement with independent human ratings and its repeated-rating consistency are not established here. Attacking the steered model also does not test whether the evaluator itself can be redirected by adversarial text in the response.
The architecture gives a concrete separation property: decoder tokens cannot write back into the encoder representation. The broader claim about behavior requires additional evidence, because the decoder still reads untrusted context and predicts future conditioning from that context. I would distinguish direct changes to the control state from changes in how the decoder responds to a fixed state. The present tests support persistence under specified counter-pressure, but not an invariant that arbitrary contextual instructions cannot change useful behavior.
The evaluator also deserves a separate validation argument. An automated concept score is useful for a broad sweep, yet it can favor responses that repeatedly invoke the concept without answering the question well. The paper’s fluency and instruction-following axes partly address this, and the response examples help. A small blinded human check on representative successes and failures, together with repeated ratings and a targeted evaluator challenge, would give clearer evidence that the observed control is not peculiar to one judge or scoring prompt.
-
The capability cost is large enough to affect the practical interpretation. ChatCORE falls from 0.3454 to 0.1998 at 3B and from 0.2212 to 0.1543 at 400M. The oracle result uses benchmark answers and an earlier model variant. It shows that the channel can carry useful information, but it does not establish that feature prediction alone causes the loss or that improving it will recover the missing capability. I would frame the current result as a control–capability tradeoff.
The oracle-conditioning experiment is informative but must remain a diagnosis, not a capability result: its signal is derived from benchmark answers and is applied to an earlier variant. It establishes that an informative channel can change prediction, while leaving open how much of the ordinary-generation loss comes from feature prediction, the training distribution, splicing or resource allocation. The paper’s proposed explanation is plausible, but the experiment does not identify it uniquely.
I would also like the cost comparison to include the data-processing burden as well as inference. The authors already report a large annotated dataset footprint and substantial preprocessing compute. That disclosure is valuable. It makes the practical question more precise: for a given training and deployment budget, how much useful control is gained after accounting for the encoder, signal construction and capability loss? This need not be resolved at frontier scale to merit publication, but it should bound the claims about scalability and inexpensive steering.
Overall Assessment
Presentation: The architecture and evaluation are clearly explained.
I particularly appreciate the explanation of why training with target-derived features creates a useful dependency while context-derived features at inference do not provide the same information advantage. That makes the central design easier to assess.
Informativeness: The pretraining approach provides an interesting alternative to post-hoc steering.
Soundness: The threat model and the interpretation of the capability loss need a narrower account.
I lean toward acceptance because the training design is creative and conceptually coherent, and the controlled small-model results provide a useful proof of concept. My main concern is the strength of the practical claim. The model is more controllable on these tests, but also less capable. I would be more convinced by an advantage at comparable useful quality and by a clearer boundary around the attacks the paper covers.
I would not require this first demonstration to match the strongest available model on every capability benchmark. The creative mechanism and the controlled small-scale evidence can merit acceptance even with a substantial limitation. What matters is presenting that limitation honestly and showing that the steering effect remains useful within a clearly defined regime. I encourage the authors to distinguish that credible contribution from a general claim of tamper-resistant or universally robust control.
Questions for authors
-
How does the advantage change under a stronger adaptive suffix search and repeated generation and attack seeds? Can blinded human ratings, repeated judge evaluations and a targeted attack on the evaluator confirm the result? Please specify what the attacker can access, especially the control channel.
-
At comparable useful-task accuracy or response quality, how much control advantage remains? What do the existing ablations tell us about the separate contributions of conditioning, feature prediction, and splicing?
-
How does this approach differ conceptually from a mixture-of-experts model whose router is conditioned on the same semantic features? Conditioning appears to encourage specialization across kinds of text, although your model shares decoder parameters rather than routing among separate expert subnetworks. Is the important distinction the shared representation, the separate control channel, or the training incentive to use it? Would a feature-conditioned MoE be expected to have similar control properties, or is there a mechanism specific to your design?
-
How sensitive are the main conclusions to the selected steering strength and to the quality threshold at which the models are compared? Can the existing sweeps identify a region where control remains stronger without relying on a substantially less capable or less fluent operating point?
-
What evidence separates the effects of next-segment feature prediction, splicing and the separate-channel architecture? For the MoE analogy, I am asking for a conceptual distinction rather than requiring a new large-scale MoE training experiment: what property of the shared conditioned decoder is essential to the claimed control mechanism?
2 comments
RE: Signal-conditioned mixture-of-experts, and whether the important distinction is the shared representation, the separate control channel, or the training incentive
Our view is that the most important distinction between our method and feature-conditioned MoE routing is the usage of mutual information to bake steering into the next-token prediction objective. The conditioning information is passed explicitly rather than being reinforced implicitly via targeted weight updates to specific experts. However, we also believe the other two differences to be important mechanisms in our design. The separate control channel operates through one-way cross attention that ensures a consistent, unchanging set of conditioning KV vectors. We also have a larger space to optimize for these signals, as the conditioning information is passed to the whole network rather than a specific conditional weight subset.
RE: Sensitivity of model quality to steering strength
We find a drop-off at steering strengths below for our method. Despite the high steering strength that our model is effective in, it retains strong fluency even at a steering strength of 1. Pythia is unable to steer effectively without losing capabilities; LoRA and prompting retain fluency but are weaker than our method, while steering vectors are stronger (but still significantly weaker than our method) and completely destroy its language modelling ability.
We thank the reviewer for their time and detailed feedback. We appreciate that the reviewer found the central design “insightful and creative” and the choice to make the channel valuable to prediction “the most compelling part of the paper,” and that they credited the two 400M comparisons and our reporting of absolute scores, fluency, and capability loss. We agree with the reviewer’s view of this work as a proof of concept for a learned control interface.
On capabilities, the response to all reviewers describes what we found and the updated results. We agree with the reviewer that the oracle result is a diagnosis rather than a capability result. We emphasize that this is the first implementation of this approach. We believe it has demonstrated the potential of the method, and we expect that further experimentation and attention, which this paper justifies, will determine how much of a capabilities drop is necessary. Our results have already shown substantial improvement, and we believe there is more to come to close the gap.
We apologize that our threat model was not clear in the initial submission and value this feedback. We will update the paper to fix this issue. As we stated in our response to all reviewers, our threat model is that the attacker controls input tokens but not internal activations, weights, or the steering signal. We have also added additional attacks to our evaluation. Beyond the original GEPA suffix search, we now train an attacker LM to write a 64-token suffix (RL red-teaming, Sec. 5.2, App. E) and give the attacker gradient access to optimize 4 continuous suffix tokens (soft-prompt attacks, Sec. 5.3, App. F). Mancelle-3B retains 97% [92, 102] of its concept score under the RL attack and 81% under the strongest white-box attack; the comparison to baselines is in the response to all reviewers. We do not claim robustness to attacks that write directly into the model’s internals (App. F.4, G).
This is a good observation and we agree it is a potential source of vulnerability: during self-conditioned generation, the features for the next segment are selected from αfsteer+(1−α)fnext, where fnext is predicted from context, so this is a potential failure mode. Our current evaluations do not directly evaluate this. We will clearly discuss it in the limitations section of our paper.
As for the strength and rigor of the adversarial robustness evaluations, we highlight the soft-prompting and RL red-teaming attacks we have added to the paper (Sec. 5.2-5.3). Our robustness evaluations were relatively expensive due to the amount of API calls and inference steps needed (especially the RL red-teaming experiment), so we rely on our population of steering concepts to provide us with confidence intervals.
Thank you for the feedback. We agree that this is important to address and will update the paper accordingly. Since the initial submission, we had three human annotators label 150 samples from our experiments according to concept relevance, instruction following, and fluency to measure LLM judge agreement. We find agreement rates of 60%, 53%, and 42% between the LLM judge and human annotators. For comparison, the inter-annotator agreement rate for humans was 54.4%, 35.3%, and 58.8%, respectively. Additionally, we observe a monotonic increase in the average human annotator rating as LLM judge ratings increase—increased LLM judge ratings correspond to increased human judge ratings. This gives us confidence that this methodology highlights semantically meaningful differences between steering methods. We will update the paper to discuss these results.
We see competitive steerability against the much stronger Qwen2.5 and SmolLM3 baselines and much stronger steering robustness, as indicated in the answer to the previous question. For splicing, we have added an ablation to Appendix K where we train a 400M model using the existing recipe except for pretraining data splicing, showing a moderate but significant effect. For conditioning, we point to our baseline decoder-only models, which were trained with no conditioning signals, matching our model in number of total parameters and decoder parameters. For feature prediction, we do not have an ablation available, as this would require another sizable full training run.