acceptodds
Desk rejected at ICLR 2027

Value Safety under Hidden Controllers: Behavior, Specification Exposure, and Residual Interventions

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural-language controllers allow assistants to follow application-specific value priorities, but the instructions can also surface in generated responses. We study value safety as the joint requirement of preserving value-directed behavior while limiting exposure of the internal specification. We introduce a training-free interventions for Value Residual Guidance (VRG), which uses the logit difference between controller-conditioned and base passes through model. VRG-Exact zeroes selected controller-associated residual coordinates and VRG-Soft shrinks them; both retain the base logits rather than banning tokens. Experiments show that stronger guidance can increase literal exposure even when additional behavioral gains are small. Controller text also appears in non-extraction tasks, and intervention effects vary across attack families, models, and steering directions. Additional evaluations show small incremental changes in content safety and task capability for VRG-Soft relative to VRG. Our findings support a partial separation between intended behavioral influence and literal specification exposure, while semantic and functional signals of the configured value remain observable.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Value Safety, Value Steering, Controller Disclosure, Large Language Models, Inference-Time Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u8ReOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates how to reduce the exposure of internal value specifications while preserving value directed behavior in large language models guided by internal natural language value controllers. The authors formulate this problem as a trade-off between preserving value directed behavior and limiting specification exposure. To address this challenge, they propose Value Residual Guidance (VRG), a framework that leverages the logit residual between two inference passes, with and without the value controller, to reduce literal specification exposure while retaining as much value-guided behavior as possible. Experiments across multiple datasets and models demonstrate that the proposed approach can reduce literal specification leakage to some extent while largely preserving value directed behavior.

Critical strengths

1. Novel and practically meaningful problem formulation.

The paper addresses an interesting and practically relevant problem. Ensuring that language models follow intended value priorities without unnecessarily exposing internal prompts is valuable, particularly for reducing potential attack surfaces. Moreover, system prompts may contain commercially valuable information that warrants protection. The proposed approach therefore has practical potential in these settings.

2. Simple and intuitive method design.

The proposed method is straightforward from an engineering perspective. By adjusting the model's output distribution based on the difference between predictions with and without the value controller, the approach provides an intuitive mechanism for steering model behavior. Compared with approaches such as LoRA-based fine-tuning, it also avoids modifying model parameters, thereby reducing potential risks of degrading the model's original capabilities through training. In addition, the authors report relatively low computational overhead for their proposed intervention.

3. Relatively comprehensive experimental evaluation with transparent reporting of limitations.

The authors conduct a reasonably broad set of experiments to evaluate the effectiveness of their approach and openly discuss its limitations. Beyond full-text recovery rates, they examine semantic exposure, functional policy inference, and value-directed behavior. They further analyze the effects of different guidance strengths, value priorities, attack families, and models. The experiments reveal an interesting phenomenon: as guidance strength increases, improvements in value-directed behavior may begin to saturate, while specification exposure continues to increase.

Critical weaknesses

W1. The construction of the key intervention set LL is unclear, limiting the reproducibility of the proposed method.

Both VRG-Exact and VRG-Soft rely on a predefined controller-associated token set LL. However, Sections 4.2 and 5 only state that this set is fixed before formal generation and contains 51 or 49 token IDs, without clearly explaining how these tokens are selected.

W2. Lack of sensitivity analysis for key hyperparameters and the intervention set.

The authors fix the shrinkage parameter of VRG-Soft at β=0.25\beta=0.25, corresponding to retaining 80% of the selected residuals. Meanwhile, the size of LL is fixed at 51 or 49 tokens. However, the paper does not justify these specific choices or investigate how different parameter settings affect the results. Since β\beta directly controls the degree of residual shrinkage, while ∣L∣|L| determines the scope of intervention, both may influence the trade-off between specification exposure and value-directed behavior. In particular, VRG-Soft and VRG-Exact can be viewed as different intervention strengths within the same residual-scaling framework.

W3. Insufficient statistical uncertainty analysis to support some of the experimental conclusions.

The paper reports average results for behavioral separation, specification exposure, content safety, and task capability, but provides limited confidence intervals and statistical significance analyses for differences between methods. In particular, the claims that VRG-Soft preserves value-directed behavior and that residual interventions introduce only small incremental changes in content safety and task capability would benefit from stronger statistical evidence. The caption of Table 2 explicitly states that Appendix B.3 provides numerical intervals, but these intervals are not actually reported in Appendix B.3.

W4. Insufficient baseline comparisons, particularly with prompt-based approaches and other inference-time intervention methods.

The paper primarily compares the proposed approach against the original VRG, random masking, output filtering, and LoRA-based fine-tuning methods. However, it lacks a simple yet practically relevant prompt-based baseline. For example, one could directly instruct the model through a natural-language controller to follow the specified value priorities without reproducing or describing the internal instructions, without introducing additional logit residual editing.

Furthermore, although the Related Work section discusses several inference-time steering techniques, the paper does not provide sufficiently comprehensive comparisons with these approaches in terms of value-directed behavior, specification exposure, and inference efficiency. The current efficiency results mainly demonstrate that VRG-Soft introduces little additional overhead relative to the original VRG, rather than establishing that the overall two-pass framework is more efficient than single-pass inference or other inference-time control methods.

Questions for authors

Q1. The theoretical motivation behind the minimum-change interpretation.

Section 4.2 interprets VRG-Exact and VRG-Soft as Euclidean projection and quadratic regularization problems in Equations (7) and (8), respectively. While both formulations are mathematically valid, the paper does not sufficiently explain the motivation for choosing these particular optimization objectives or establish a theoretical connection between minimum changes in the residual space and the preservation of value-directed behavior or suppression of specification exposure. Could the authors clarify whether these formulations are intended merely as equivalent optimization characterizations of the predefined interventions, or whether they are meant to provide a stronger theoretical justification for the behavior–exposure trade-off?

Q2. Incomplete sentence in the threat model description.

In Section 5, under "Threat Model and Attack Protocol," the authors write: "In the black-box setting, no information about the intervention is provided; in the defense-aware setting." This sentence appears to be incomplete.

2 comments

Dear Reviewer u8Re

We hope this message finds you well.

As the rebuttal phase approaches its conclusion, we would like to sincerely thank you once again for your time and thoughtful comments. We submit detailed responses addressing your concerns a few days ago, and we hope they have resolved the issues you raised.

If there are any remaining questions or clarifications needed from our side, please do let us know — we would be more than happy to respond promptly.

Otherwise, if you feel our responses have sufficiently addressed your points, we kindly ask you to consider updating your review accordingly.

We greatly appreciate your contribution to improving our work.

Warm regards.

I appreciate the authors' detailed rebuttal and the additional experimental results. The responses have addressed most of my major concerns, particularly regarding the construction of the intervention set, hyperparameter sensitivity, and baseline comparisons. Therefore, I have decided to increase my soundness rating.

Reviewer_2Xr7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how language models can follow hidden value instructions without revealing them. The authors propose two training-free decoding methods, VRG-Exact and VRG-Soft, which selectively modify controller-induced logit changes to reduce information leakage. Experiments across different values, attack settings, and models show that these methods can reduce verbatim exposure of internal instructions while largely preserving value-guided behavior. However, the underlying value preferences remain observable through the model's responses.

Critical strengths
  • The method requires no additional training or external detectors and can be applied directly at inference time. Although technically simple, it offers a practical way to reduce controller text exposure while preserving value-guided behavior.

  • The experiments suggest that value-guided behavior and controller text exposure can be partially separated. In the main Qwen experiments, VRG-Exact substantially reduces verbatim recovery of the controller text while retaining most of the value-guided behavioral effect.

Critical weaknesses
  • The paper does not explain how the key token set LL is constructed or specify the models used as automatic evaluators, making the experiments difficult to reproduce.
  • Some reported results are inconsistent. For example, the XSTest results in Table 7 do not match those in Appendix Tables 11 and 12, and the average reported for Gemma in Table 6 does not agree with the values for the two steering directions. These issues raise concerns about the reliability of the experimental results.
  • Value-guided behavior is evaluated mainly using automatic judges and machine-generated action anchors, without independent human validation. It is therefore unclear whether the interventions genuinely preserve the model's decision-making behavior.
  • The key Qwen VRG-Exact experiment lacks a corresponding human evaluation of specification leakage. The current results mainly demonstrate a reduction in verbatim text recovery, but do not establish whether the model can still reveal internal instructions through paraphrasing or descriptions.
  • The random-mask baseline uses zeroing, while VRG-Soft uses scaling. This is not a well-controlled comparison and makes it difficult to assess whether the selected token set is actually effective.
Questions for authors
  • What is the exact procedure for constructing LL, and what is the full token list? Is LL constructed separately for each value, or is it the union of tokens from multiple controllers?

  • Could the authors clarify the discrepancies in the XSTest results between Table 7 and Appendix Tables 11 and 12, the Gemma macro-average in Table 6, and the mismatch between the 591 samples summed across values in Table 3 and the 589 action anchors reported in the main text?

1 comment

Dear Reviewer 2Xr7

We hope this message finds you well.

As the rebuttal phase approaches its conclusion, we would like to sincerely thank you once again for your time and thoughtful comments. We submit detailed responses addressing your concerns a few days ago, and we hope they have resolved the issues you raised.

If there are any remaining questions or clarifications needed from our side, please do let us know — we would be more than happy to respond promptly.

Otherwise, if you feel our responses have sufficiently addressed your points, we kindly ask you to consider updating your review accordingly.

We greatly appreciate your contribution to improving our work.

Warm regards.

Reviewer_N51jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free method to reduce the direct exposure of hidden instructions while preserving value-directed behavior. The method shows promising results across multiple models and datasets.

Critical strengths
  1. The problem is practically relevant, as preventing system prompt leakage is an important concern in real-world LLM applications.

  2. The proposed method is training-free and relatively easy to implement, while achieving reasonably good results.

  3. The evaluation is fairly comprehensive, covering different models, values, and extraction attacks. The authors also examine the method's impact on unrelated tasks, including general capabilities and content safety.

Critical weaknesses
  1. Lack of simple but potentially strong baselines. The paper does not compare against some straightforward alternatives, such as explicitly instructing the model not to reveal its internal instructions in the controller itself. Such baselines could already be quite effective and would help establish the actual benefit of the proposed method.

  2. Inconsistent effectiveness across models. The results are not always consistent across models and steering directions. For example, VRG-Exact increases macro literal recovery by 2.08 percentage points on Llama and 3.25 percentage points on Gemma at (λ=0.5\lambda=0.5). This suggests that the effectiveness of the method depends heavily on the model, steering direction, and operating point.

  3. High computational overhead. The method requires two forward passes at every decoding step, roughly doubling the inference computation compared to standard generation. This additional cost could limit its practicality in real-world applications.

Questions for authors

See weakness

1 comment

Dear Reviewer N51j

We hope this message finds you well.

As the rebuttal phase approaches its conclusion, we would like to sincerely thank you once again for your time and thoughtful comments. We submit detailed responses addressing your concerns a few days ago, and we hope they have resolved the issues you raised.

If there are any remaining questions or clarifications needed from our side, please do let us know — we would be more than happy to respond promptly.

Otherwise, if you feel our responses have sufficiently addressed your points, we kindly ask you to consider updating your review accordingly.

We greatly appreciate your contribution to improving our work.

Warm regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.