acceptodds
Under review as a conference paper at ICLR 2027

Treating Alignment as an In-Context Approximation Problem

Abstract

Alignment changes model weights, yet much of what steers model behavior at test time happens in context. Their immediate relationship is so far little understood. We study when context-induced distribution shifts can reproduce those induced by parameter updates. We first show that, in linear attention, a fixed KV prefix exactly reproduces a low-rank bilinear update, and characterize where this fails, including updates whose effects depend on task content. This motivates task-dependent context, where relevant clauses are retrieved from an approved constitution. We identify three conditions for close approximation: the retrieved clauses fit the update, the frozen model realizes their declared effects, and the clauses retain those effects when combined. Controlled experiments agree with this picture: across LoRA updates, retrieving the relevant clause recovers the trained rule in frozen Qwen3 models and brings them much closer to their aligned counterparts, while presenting the full constitution can introduce errors from irrelevant and competing rules. Finally, we identify two fundamental limits of in-context alignment. Exact bounded clause selection is NP-complete in the worst case, and some training-induced shifts lie in directions that no deterministic context can follow.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, retrieval, LLMs, constitutional AI, in-context learning, prefix tuning, learning theory, low-rank approximation, sparse recovery

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fx3kOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores a connection between fine-tuning and in-context learning. They show that having a certain library of sources and a retriever, and using some of the retrieved content in the prompt, can theoretically recover a fine-tuned model whose parameters have changed.

Critical strengths
  • I believe this paper offers what a lot of other submissions do not, and that is clarity. The paper is clear to read, and the illustrative figure helps a lot. The math is clear and intuitive, and the terms defined, like induced shifts and oscillation, are well-defined. The assumption about bilinear decomposition is clearly stated. The authors also mention the failure cases.
  • The theoretical result, specifically Proposition 1, carrries not only mathematical rigor but also key insights, shown by the the three terms that outline how to inform in-context alignment, namely the fitness of the clause, the relaization of the inted effects of those clasues, and divesity of the rules.
  • the experiemnts give a hint towards the validity of the method.
Critical weaknesses
  • The experimental results are limited, and I cannot exactly capture the benefits of this method from one example mentioned in the figures, especially while using an oracle context. We would be more interested in a more realistic context that is not guaranteed to be perfect. I also think we need more diverse datasets and applications.
  • The theoretical results have strong assumptions about the linearity of the attention heads.
  • I don't think the lietratre was sufficiently addressed, and I would liek to know how the paper's proposed problem is different from what is in [1], [2], [3], [4], [5], [6].

[1] https://aclanthology.org/2023.findings-acl.779/

[2] https://aclanthology.org/2026.acl-long.1932/

[3] https://proceedings.iclr.cc/paper_files/paper/2024/hash/18a367688479c1b08b001584218a4443-Abstract-Conference.html

[4] https://aclanthology.org/2024.findings-emnlp.182/

[5] https://arxiv.org/abs/2602.21103

[6] http://proceedings.iclr.cc/paper_files/paper/2026/hash/ff6f1a8c167f5f17a32981ecae4f3539-Abstract-Conference.html

Questions for authors
  • Between fine-tuning and in-context learning, there is a family of methods that do inference-time alignment without modifying the weights of the model, as in controlled decoding [1], [2], [3]. Can we recover the same performance of controlled decoding using in-context learning as well?
  • How much overhead does the retriever addition add to the inference latency?

[1] https://arxiv.org/abs/2310.17022

[2] https://arxiv.org/abs/2405.20495

[3] https://arxiv.org/abs/2505.23729

6 comments
AuthorsOpenReview ↗

Empirical Validation of the Approximation Error Decomposition: Bound Tightness, In-Context Safety Alignment, and Constitutional On-Policy Self-Distillation

TL;DR

The reviewer appreciates both the clarity and insight provided by the core result of our paper (approximation error decomposition), but also requests more empirical evaluations to establish the results’ explanatory and predictive value. To this end, we have expanded our experimental evaluation, which now comprises three complementary parts:

1. Relationship between context and training updates. We extend the experiment from our initial submission, illustrating the overall relationship between fine-tuning and context-induced distribution shifts with an experiment to measure the tightness of the proposed theoretical bound and the impact of the different error terms.

2. Explanatory power in realistic alignment. We isolate the effects of the three approximation errors on a held-out SafeChain split and show that controlling them allows context-induced performance to approach, and under privileged information even exceed, the corresponding SFT performance in safety alignment.

3. Connection to on-policy self-distillation. We extend our theoretical analysis to characterize when self-distillation can recover a desired target policy despite the context-induced teacher only approximating it. This establishes a direct connection to our error decomposition and explains how controlling the individual errors can improve constitutional self-distillation. For more details on the theory see here: https://openreview.net/forum?id=FDitFfgFgK&noteId=YozD0dRtL9.

Below we summarize the main new experimental results, which will all be included in the revised paper.

New empirical results

1. The empirical tightness of the theoretical bound on DRD_R.

We empirically measure the three terms of our approximation error decomposition and find that the proposed upper bound becomes increasingly tight with model performance, supporting its informativeness. For Llama-3.1-70B and Qwen3.8-27B the bound lies 0.3 to 0.6 log-odds above the measured error. For comparison, in our experimental setup, a model at chance is 2.94 away from the correct answer. Thus, the bound is tight for strong models. For more details on the experiment, see here: https://openreview.net/forum?id=FDitFfgFgK&noteId=C1Jq0W8gcn.

2. Approximation error decomposition in real alignment with imperfect context

We additionally observe the impact of all three approximation errors on a held-out SafeChain split. All three have statistically significant effects (p < 0.001 for both model sizes): improving clause selection increases composite overrefusal and safety performance by +5.51/+3.31 pp, reducing clause interactions results in improvements of +2.52/+2.13 pp, and providing privileged interpretation support leads to +8.06/+3.34 pp higher safety performance (1.7B/8B, respectively, safety & overrefusal measured by Qwen3Guard on a SafeChain). For more details on the experiment, see here: https://openreview.net/forum?id=FDitFfgFgK&noteId=YwvXWcNlfE.

3. Explaining improved self-distillation from controlled in-context approximation

Tables 1 and 2 demonstrate the practical implications of our decomposition for on-policy self-distillation. Both Qwen3 1.7B and 8B are trained on SafeChain using teacher configurations that ablate the effects of controlling context approximation errors. All models, including the SFT baseline, are trained with a batch size of 64 and 350 training steps. We then evaluate the trained models on HarmBench, StrongREJECT, and XSTest (↓) for safety, and MMLU-Pro, MATH-500, and GPQA-Diamond (↑) for reasoning and general capabilities, all at pass@1 and the same generation parameters. The tables report the corresponding safety, reasoning, and overall composite scores. Controlling the teacher's approximation errors improves the overall composite by +4.78 pp (1.7B) and +2.55 pp (8B) compared with uncontrolled self-distillation. Crucially, the resulting policies also outperform the corresponding SFT baselines by +2.85 pp and +2.93 pp, respectively. For more details on the experiment, see here: https://openreview.net/forum?id=FDitFfgFgK&noteId=YwvXWcNlfE.

Table 1. Qwen3-1.7B: Performance after self-distillation.

MethodSafety (%) ↑Reasoning (%) ↑Overall (%) ↑Δ vs pretrained (pp)Δ vs SFT (pp)
Naive self-distillation without context approximation error control83.2046.5964.89+3.21-1.93
Self-distillation with context approximation error control in the teacher91.36*47.9869.67*+7.99+2.85

Table 2. Qwen3-8B: Performance after self-distillation.

MethodSafety (%) ↑Reasoning (%) ↑Overall (%) ↑Δ vs pretrained (pp)Δ vs SFT (pp)
Naive self-distillation without context approximation error control91.8163.3177.56+1.71+0.38
Self-distillation with context approximation error control in the teacher95.95*64.2880.11*+4.26+2.93
AuthorsOpenReview ↗

Linear Attention Assumption

The reviewer remarks that our theoretical results have strong assumptions about the linearity of the attention heads. However, the core of our results, i.e., the decomposition of the approximation error and its connection to self-distillation, does not assume linear attention heads. The results hold for general transformers. Only the KV-prefix section does make the linearity assumption, and it serves as theoretical motivation rather than as a main contribution. To make this clearer, we have moved most of the KV-prefix section, including its attention-linearity assumption, to the appendix. The main text keeps only its key findings, which motivate approximating training updates with clauses in context. We thank the reviewer for bringing this to our (linear) attention.

AuthorsOpenReview ↗

More diverse datasets and applications

The reviewer requests evaluations on more diverse datasets and applications than the initial synthetic setting. In addition to our initial dataset, we introduce one more synthetic setup (logic riddles) as well as a realistic alignment setup that involves training on SafeChain and evaluating on a SafeChain hold-out set, as well as HarmBench, StrongREJECT, and XSTest (↓) for safety, and MMLU-Pro, MATH-500, and GPQA-Diamond (↑) for reasoning and general capabilities.

Thus, we have expanded our investigations from one to eight datasets spanning harmfulness, overrefusal, reasoning, general knowledge, and logical in-context learning evaluations.

Crucially, we observe that the explanatory power of our theory holds across the tested settings, as detailed here: https://openreview.net/forum?id=FDitFfgFgK&noteId=If8lLMbdGL.

Reviewer_ma5yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates when context-based interventions can approximate the behavioral changes induced by alignment training. It establishes theoretical connections between weight updates and KV prefixes, proposes an error decomposition based on clause selection, interpretation, and interaction, and identifies fundamental limitations of in-context alignment. Experiments on synthetic alignment tasks show that retrieving relevant policy clauses can closely reproduce fine-tuned model behavior, while irrelevant or competing clauses can degrade performance.

Critical strengths

The paper addresses an interesting and meaningful question about the relationship between in-context and parameter-based alignment. Its main strength is the relatively rigorous theoretical analysis, with clearly stated assumptions, detailed proofs, and formal verification of several results. The controlled experiments also provide useful insights into how irrelevant or competing clauses can interfere with contextual alignment.

Critical weaknesses

1. Limited significance and generalizability of the empirical results

The empirical evaluation is largely restricted to synthetic tasks where fine-tuned models learn associations between invented diseases and doctors, and the frozen model is provided with an explicit clause specifying the exact desired response. This setting is considerably simpler than realistic alignment, which typically involves nuanced behavioral preferences, safety judgments, and context-dependent decisions. In particular, the strong performance of oracle clause retrieval is somewhat expected, as the frozen model only needs to follow an explicit instruction containing the target answer. Therefore, it is unclear whether these experiments reveal meaningful insights about approximating training-induced alignment shifts, beyond demonstrating that instruction-following models can reproduce simple behaviors when given the corresponding rules. More realistic alignment tasks are needed to substantiate the paper's broader motivation and claims.

2. Weak connection between the theoretical analysis and empirical validation

Although the paper develops several theoretical results, their relevance to the experiments remains insufficiently established. The KV-prefix approximation theorem relies on a single unnormalized linear-attention head and an assumed bilinear decomposition of weight-induced shifts, but the experiments use modern transformers with softmax attention and natural-language prompts. The applicability of these results to practical alignment updates is not investigated. Moreover, Proposition 6 decomposes approximation error into clause-fit, interpreter, and interaction terms primarily through an algebraic identity and the triangle inequality. The experiments do not quantitatively measure these terms or demonstrate that they predict the observed approximation errors. The authors also acknowledge that the experimental clause effects do not satisfy the scoped rank-one assumption (Appendix F.5). Consequently, the experiments largely provide qualitative illustrations rather than direct validation of the proposed theoretical framework.

3. Limited technical novelty and insufficient differentiation from closely related work

Several theoretical contributions appear to be relatively direct applications of existing mathematical insights. The KV-prefix rank characterization follows from the additive structure of unnormalized linear attention and standard low-rank matrix factorization, while the retrieval hardness and frozen-readout impossibility results build on classical NP-completeness and the established softmax bottleneck, respectively. Although combining these observations within an alignment-approximation framework is potentially interesting, the paper does not sufficiently establish what fundamentally new understanding this framework provides beyond prior work on the equivalence and limitations of contextual versus parametric adaptation (e.g., Petrov et al., 2024; Chen et al., 2024; Goldwaser et al., 2025). Furthermore, existing in-context alignment studies (e.g., Lin et al., 2024; Zhao et al., 2025) have already investigated the ability of prompting to approximate fine-tuned behavior on more realistic tasks. Without stronger empirical comparisons or novel predictions derived from the theory, the incremental contribution over these studies remains unclear.

Questions for authors

see above.

6 comments
AuthorsOpenReview ↗

Demonstrating the explanatory power of the approximation error decomposition on realistic alignment tasks

TL;DR

The reviewer requests testing the explanatory power of our theory beyond controlled settings. We agree that synthetic association tasks alone cannot establish the relevance of our error decomposition to realistic alignment. To address this, we extended our evaluation to safety alignment on SafeChain, HarmBench, StrongREJECT, and XSTest while assessing reasoning retention on MMLU-Pro, MATH-500, and GPQA-Diamond using Qwen3-1.7B and Qwen3-8B. As a constitution, we adopt Anthropic's constitution covering a set of 56 clauses (Full Constitution) with a subset of 14 clauses relevant for safety alignment (Safety Constitution). Responses are evaluated with Qwen3Guard and WildGuard. The experiments yield two key findings:

1. Reducing approximation errors makes in-context alignment match or exceed the SFT baseline on held-out SafeChain. Addressing selection, interaction, and interpretation errors yields gains of +5.51 / +3.32, +2.52 / +2.13, and up to +8.06 / +3.34 percentage points, respectively (1.7B / 8B). All improvements are statistically significant (two-sided permutation test, 100,000 permutations; p < 0.001).

2. Controlling these errors during training significantly improves overall constitutional on-policy self-distillation outcomes (p < 0.001). Adjusting the teacher's context to reduce all three approximation errors yields +4.78 / +2.55 pp over naive constitutional self-distillation and +2.85 / +2.93 pp over the corresponding SFT baselines (1.7B / 8B).

All results and implementation details will be included in the revised paper.

1. The effects of approximation errors in alignment

We first evaluate pure in-context alignment on the held-out SafeChain split, without any parameter updates. This allows us to isolate the effects of the approximation errors on context-induced policies through controlled comparisons:

  1. Selection: We compare the Safety Constitution (14 rules) against a Partial Constitution containing six safety and eight other rules of the Full Constitution (also 14 rules).
  2. Interaction: We compare the Safety Constitution with the Full Constitution.
  3. Interpretation: We introduce privileged interpretation support to help the model in detecting when and how to apply the rules in the constitution: Support 1 supplies the safety label, while Support 2 additionally specifies the corresponding response behavior.

Our findings show significant effects for all three error sources:

Composite safety performance (Harm & Overrefusal) (%) on held-out SafeChain.

Primary error sourceContext1.7B (%)8B (%)1.7B Δ vs SFT (pp)8B Δ vs SFT (pp)
SelectionPartial Safety (6 safety + 8 other)76.0287.83−13.12−3.74
Safety Constitution (14 safety)81.53***91.15***−7.61−0.43
InteractionFull Constitution (56 clauses)79.0189.02−10.13−2.55
Safety Constitution (14 clauses)81.53***91.15***−7.61−0.43
InterpretationSafety Constitution81.5391.15−7.61−0.43
+ Interpretation Support 184.77***94.49***−4.37+2.92
+ Interpretation Support 289.59***94.45***+0.44+2.88

2. Connecting approximation errors to on-policy self-distillation

Although privileged interpretation support, such as a safety label, is generally unavailable at deployment, it can be supplied to the teacher during on-policy self-distillation. Motivated by our new theoretical extension connecting teacher approximation error to self-distillation, we compare naive constitutional self-distillation against a teacher using Safety Constitution + Interpretation Support 2, designed to address all three error sources.

Both the SFT baselines and the self-distillation students are trained on SafeChain using the same batch size of 64 and 350 training steps. We evaluate the resulting trained models without constitutions or privileged information on HarmBench, StrongREJECT, XSTest, MMLU-Pro, MATH-500, and GPQA-Diamond, reporting higher-is-better composites.

Qwen3-1.7B

MethodSafety (%) ↑Reasoning (%) ↑Overall (%) ↑Δ vs SFT (pp)Δ vs pretrained (pp)
Naive self-distillation83.2046.5964.89−1.93+3.21
Self-distillation with teacher approximation error control91.36***47.98*69.67***+2.85+7.99

Qwen3-8B

MethodSafety (%) ↑Reasoning (%) ↑Overall (%) ↑Δ vs SFT (pp)Δ vs pretrained (pp)
Naive self-distillation91.8163.3177.56+0.38+1.71
Self-distillation with teacher approximation error control95.95***64.2880.11***+2.93+4.26

Significance level: * p < 0.05, *** p < 0.001

AuthorsOpenReview ↗

W2: Prefix theorem resides on linear attention assumptions

TL;DR

The reviewer points out that our KV-prefix approximation theorem relies on standard theoretical simplifications, such as linear attention, which limit its direct applicability to modern transformers. We acknowledge this limitation but emphasize that our core theoretical contributions—the approximation error decomposition and its newly introduced connection to on-policy self-distillation (Proposition 8 introduced above)—do not rely on these simplifying assumptions. To better reflect this distinction and sharpen the focus on the paper's main contributions, we have moved the detailed KV-prefix analysis to the appendix, retaining only its theoretical motivation in the main text.

Additionally, we introduce a new KV-prefix experiment with Qwen3-8B in the appendix, demonstrating that learned KV prefixes can closely approximate fine-tuned model behavior on the prompts used for fitting, even in a modern transformer with softmax attention.

While this does not establish the precise rank-to-rank correspondence of the simplified theorem, it provides a complementary empirical illustration that the underlying intuition extends beyond the theoretical setting.

Number of distinct target replies learned by the updatePrefix lengthTV distance
1 (shared reply)10.008–0.030
1610.101
6410.255
64160.026
AuthorsOpenReview ↗

W3: Contribution beyond related work

TL;DR

The reviewer recognizes the value of studying the relationship between alignment updates and in-context learning, but requests a clearer differentiation from the cited literature. While previous work has established mathematical and empirical characterizations of in-context shifts and weight updates, none provide an upper bound on the in-context approximation of weight updates that decomposes into easily interpretable terms, nor do they demonstrate the informativeness of the bound and the explanatory value of the terms for informing better in-context alignment. Crucially, neither does existing work establish a connection to constitutional on-policy self-distillation. We provide this link both theoretically and empirically and hence give the topic of in-context alignment a new perspective as a crucial mechanism during constitutional self-distillation (see here: https://openreview.net/forum?id=FDitFfgFgK&noteId=YozD0dRtL9).

To facilitate a clearer focus in the main text on this contribution, we have resolved to move most of the hardness and kv-prefix results to the appendix. We thank the reviewer for helping us with this prioritization.

What prior research establishes

The cited studies provide important results on the expressive capabilities and limitations of prompting, mathematical connections between contextual conditioning and parameter updates, and the empirical successes and limitations of in-context alignment. These findings are directly relevant to our investigation, particularly our auxiliary results on KV-prefix approximation and the general idea of in-context alignment.

Our distinct contributions

Our central contribution, however, goes beyond these established results. We connect in-context alignment to constitutional on-policy self-distillation, deriving an approximation error decomposition into clause-selection, interpretation, and interaction effects while establishing their theoretical importance for better self-distillation training outcomes. Neither this connection nor the error decomposition is established by the cited literature.

To demonstrate the practical implications of our theoretical analysis, we have added two sets of experiments:

  1. Approximation error analysis. We measure all three error components and evaluate the tightness of our theoretical upper bound against the observed approximation error. The bounds are particularly tight for models with stronger rule interpretation. For Llama-3.1-70B and Qwen3.8-27B, they exceed the measured error by only 0.3–0.6 log-odds units, compared with a chance-level error of 2.94 in our setup (for more details, see our separate response about the experiment: https://openreview.net/forum?id=FDitFfgFgK&noteId=C1Jq0W8gcn).

  2. Safety alignment through self-distillation. We demonstrate the practical benefits of this connection by adjusting the contextual teacher to address all three error sources. Across HarmBench, StrongREJECT, XSTest, MMLU-Pro, MATH-500, and GPQA-Diamond, this approach improves overall performance by 4.78 / 2.55 percentage points over naive self-distillation and 2.85 / 2.93 points over the corresponding SFT baselines (Qwen3-1.7B / 8B). More details are provided here: https://openreview.net/forum?id=FDitFfgFgK&noteId=YwvXWcNlfE.

Reviewer_1STpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when context conditioning can approximate training-induced policy changes. It connects local probability shifts to response-level error, characterizes KV prefixes in linear attention, and decomposes approximation errors from retrieved clauses. Synthetic fine-tuning experiments compare selected clauses with full constitutions. The paper also analyzes exact-selection hardness and limitations imposed by a frozen output readout.

Critical strengths
  • Comparing context conditioning with a specific training-induced policy change provides a useful perspective on prompting and fine-tuning.
  • Explicit assumptions, extensive proofs, and Lean formalizations for several central results make the analysis easier to assess.
  • Matched-length prose, unrelated rules, clause ordering, and out-of-scope questions help examine different causes of failure. The supplementary semantic-scope tests are also useful.
Critical weaknesses
  • The experiments do not evaluate the bound in Proposition 6 or instantiate the declared clause effects. It is therefore unclear whether the proposed conditions quantitatively explain the observed agreement. An explicit example showing how a referral clause fits Assumption 5 would help.
  • Most experiments use literal name matching and fixed replies. In the harder semantic setting, Table 5 shows that model-selected retrieval is farther from the target than the full constitution on trained and held-out diseases across the Qwen sizes. This finding deserves more attention in the main text.
  • Proposition 7 concerns exact selection with arbitrary encoded scopes and a variable clause budget. It does not establish hardness for approximate or fixed-budget retrieval, while the main experiments mostly select one relevant clause.
  • SPICA (Chen et al., Findings of ACL 2025) and Reflect (Bell et al., 2026) would help position the work against existing retrieved and constitution-based in-context alignment.
Questions for authors
  • Which response distribution do the bounds on p. 8 refer to, and how do they account for vocabulary truncation, EOS suppression, and sampling uncertainty?
  • How does a clause prescribing a multi-token reply instantiate Assumption 5? Do existing measurements show that the resulting bound is informative?
  • What is the principal advance over the cited context–weight equivalence results, and which existing experiment supports its relevance?
7 comments
AuthorsOpenReview ↗

W1: Measuring the terms of Proposition 6 and evaluating the bound

TL;DR: The reviewer sees value in our approximation error decomposition result, but requests an experiment to measure the three separate terms and evaluate the bound of Proposition 6. We added an experiment that conducts the requested evaluations, observing findings that affirm the proposed bound. E.g., for capable models, the bound lies 0.3–0.6 log-odds units above the measured error, compared with a chance-level error of 2.94. Below, we outline the key findings, while additional details of the experimental setup and more detailed reported tables can be found in our separate comment here: https://openreview.net/forum?id=FDitFfgFgK&noteId=C1Jq0W8gcn.

Overall findings:

  • Each controlled modification (rule depth for interpretation errors, rule-chain-depth for interaction errors, and clause selection for clause selection errors) raises its own term in almost all tested 12 models.
  • The error decomposition helps to investigate where a model fails in its in-context learning. Llama-3.1-70B reads each rule almost perfectly but fails to combine them (ι rises with the chain).
  • The tightness of our theoretical bound is closer for capable models: 0.3 to 0.6 for Llama-3.1-70B, 0.6 to 3.7 for Qwen3.5-27B, 4.8 to 27 for Llama-3.1-8B. It does not follow size, since Qwen3.8-27B, as large as Qwen3.5-27B, has 0.3 to 0.6. Weaker models make errors on single rules that cancel when the rules are combined, so the bound grows while DRD_R does not.

We thank the reviewer for inspiring us to pursue this experiment and look forward to including it in the paper.

AuthorsOpenReview ↗

Which response distribution do the bounds on p. 8 refer to

TL;DR:

The reviewer asks which response distribution we refer to on page 8. Below, we provide an answer.

Answer:

We compare the distributions of eight-token responses from the fine-tuned model and from the frozen model given the relevant clause, sampled at temperature 1 with stopping disabled, and average the distance over test questions. The reported numbers are therefore empirical estimates from sampled responses, not upper bounds. We have added corresponding clarifications to the paper to make this clearer.

AuthorsOpenReview ↗

How does a clause prescribing a multi-token reply instantiate Assumption 5?

TL;DR

The reviewer asked how a multi-token reply clause instantiates Assumption 5 and whether empirical measurements show that our approximation bound is informative. We clarified that Assumption 5 is only required for the hardness result, not for our approximation bounds or multi-token evaluation. We revised the presentation accordingly and added a worked multi-token example in the appendix. We also ran new experiments showing that the bound becomes increasingly informative and tight as model ICL capability increases. E.g., for Llama-3.1-70B, our measured upper bound exceeds the observed error by only 0.3–0.6 log-odds units (within ~⅓ of the actual error) and is substantially tighter than a chance-level upper bound of 2.94.

On Assumption 5:

Our approximation bounds and multi-token referral evaluation do not rely on the scoped rank-one restriction in Assumption 5. They accommodate arbitrary state-dependent declared effects. We use the rank-one restriction only for the NP-completeness result, which is a secondary finding and not part of the paper's core argument: the error decomposition and its new extension to on-policy self-distillation.

We acknowledge that this distinction was not sufficiently clear in the original presentation. To address this, we have revised the main text to define general state-dependent effects and moved the restricted hardness result to the appendix. We have also added a worked example in the appendix illustrating how a clause induces the intended effect in a multi-token setting (see our separate response to the reviewer’s request for such an example here: https://openreview.net/forum?id=FDitFfgFgK&noteId=P65X6dgUcx).

On the informativeness of the approximation bound for clause-based context:

To assess whether the bound is informative in practice, we conducted an additional experiment measuring the approximation error and each of the three terms in the bound, examining their trends and the tightness of the resulting bound (see here for experimental details and result tables: https://openreview.net/forum?id=FDitFfgFgK&noteId=C1Jq0W8gcn).

The results show that the bound is tight for more accurate models. For Llama-3.1-70B and Qwen3.8-27B, the bound exceeds the observed approximation error by only 0.3–0.6 log-odds units, corresponding to at most approximately one-third of the observed error. For context, the log-odds error is 2.94 for a model predicting at chance level and 5.89 for a confidently incorrect prediction. The bound is looser for weaker models. We discuss the reasons for this behavior here: https://openreview.net/forum?id=FDitFfgFgK&noteId=C1Jq0W8gcn.

Reviewer_5DZgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper explores whether, and under what conditions, an appropriate context can approximately reproduce the policy shift caused by a training update. The main contributions are as follows. First, the paper connects errors in next-token log-probability shifts to differences in complete response distributions, and characterizes when a key–value prefix exactly reproduces a bilinear update. Second, for task-specific clauses retrieved from a constitution, the authors identify theoretically and observe empirically three conditions for close approximation: the retrieved clauses fit the update, the frozen model realizes their declared effects, and the clauses retain those effects when combined. Third, the authors show that exact bounded clause selection is NP-complete and that some training-induced shift directions cannot be reproduced by any deterministic context. Experiments use LoRA fine-tunes of Qwen3 and OLMo-3 trained on synthetic rules mapping invented diseases to specific doctors, and compare the fine-tuned model with the frozen base model given the relevant clause or the full constitution.

Critical strengths
  • Strong theoretical paper on how in-context learning works.
  • Proofs are detailed and partly formalized in Lean.
  • The error decomposition (fit / interpreter / interaction) is intuitive and gives a useful vocabulary for in-context alignment.
Critical weaknesses
  • None
Questions for authors
  • How does the method behave when clauses have overlapping or conflicting scopes?
  • Is there any empirical evidence that the rank characterization of Theorem 3 is predictive in real LLMs?
3 comments
AuthorsOpenReview ↗

How does the method behave when clauses have overlapping or conflicting scopes?

TL;DR: The reviewer requests whether our theory also includes cases where the intended effects of composite rules can overlap and have non-additive dynamics. This is an excellent point, which has led to an extension of our definition of the quantity uRu_R, i.e., the intended effect of a rule set RR.

Theoretical Extensions

To accommodate the effects the reviewer mentioned, we more generally write

\qquad R = {j_1,\ldots,j_N},$$ where composition is prescribed by the rules' declarations and evaluated from left to right in the order specified by their dependencies and priorities. This allows for more complex rule compositions, of which the former additive setting, $u_R = \sum_{j\in R}u_j$, is a special case. Our theorem also holds in this setting. The approximation-error bound still decomposes into clause selection error \mathcal{A}(x,R), clause interpretation error $\sum_{j\in R}\eta_j(x)$, and clause interaction error ($\iota(x,R)$). Additionally, we distinguish between the realized interaction $I_R$ and the intended interaction $J_R$, defined as $$I_R := \delta_R - \sum_{j\in R}\delta_j, \qquad J_R := u_R - \sum_{j\in R}u_j.$$ The clause interaction error is then defined as $$\iota(x,R) := \max_{s\in\Sigma_x} \operatorname{osc}_s(I_R-J_R).$$ Here, $\delta_R$ and $\delta_j$ denote the model's realized log-probability shifts under the joint rule set and each rule individually, respectively. $\Sigma_x$ denotes the reachable states for task $x$. Intuitively, the interaction error now describes the realized interaction effect between rules that is not intended by their composition. For example, if one rule would block another, but the model failed to realize this blockage because it was distracted by another rule, this would constitute such an unintended interaction effect. ### Experimental evaluation of the error decomposition To measure all three error terms, including interaction effects under increasingly nested rule chains, and empirically examine the tightness of the predicted bound, we conduct a new experiment testing 12 models, including Llama 3.1 8B, Qwen 3.5 27B and Llama 3.1 70B as described here: https://openreview.net/forum?id=FDitFfgFgK&noteId=C1Jq0W8gcn. Overall, we observe that the true approximation error $D_R$ remains below the bound in all tested settings, as the triangle inequality guarantees, with the bound becoming tighter as model performance improves. For Llama-3.1-70B, it lies only 0.3–0.6 log-odds units above the measured error, compared with a chance-level error of 2.94 in our setup.
AuthorsOpenReview ↗

Is there any empirical evidence that the rank characterization of Theorem 3 is predictive in real LLMs?

TL;DR: The reviewer is interested in empirical evidence that the rank characterization theorem is predictive in real LLMs. Experiments on Qwen3-8B support Theorem 3’s qualitative prediction: more complex fine-tuned updates require longer KV prefixes to approximate accurately. With a one-token prefix, TV distance increases from 0.008–0.030 for one target reply to 0.255 for 64 replies, but drops to 0.026 with a 16-token prefix. However, the theorem’s quantitative bound is uninformative for real LLMs, so a rigorous rank–prefix length relationship under softmax attention remains an open question.

Number of distinct target replies learned by the updatePrefix lengthTV distance
1 (shared reply)10.008–0.030
1610.101
6410.255
64160.026
AuthorsOpenReview ↗

Connecting in-context approximation error decomposition to constitutional on-policy self distillation progress

A New Result

As the reviewer appreciated the theoretical insight of our approximation error decomposition, we think it might be of interested, that we were able to establish a further connection of this decomposition to training progress in constitutional on-policy self distillation as outlined here: https://openreview.net/forum?id=FDitFfgFgK&noteId=YozD0dRtL9.

In short: A student model in constitutional OPSD improves whenever its gain from matching the teacher exceeds the teacher's in-context target approximation error bound multiplied by how much the student changes.

Crucially, reducing selection, interpretation, and interaction errors makes this condition easier to satisfy.

Empirical Tests

We have tested the effect of this theoretical insight empirically, training models via on-policy self-distillation on SafeChain. We indeed find that under otherwise matched training and evaluation regimes, addressing the three errors in the teacher's context during on-policy self-distillation improves the resulting student policies by +4.78 / +2.55 pp over naive self-distillation and +2.85 / +2.93 pp over the corresponding SFT baselines (1.7B / 8B), with trained students evaluated without context or privileged information on HarmBench, StrongREJECT, XSTest, MMLU-Pro, MATH-500, and GPQA-Diamond:

Training Method (Qwen3-1.7B)Safety (%) ↑Reasoning (%) ↑Overall (%) ↑Δ vs SFT (pp)Δ vs pretrained (pp)
Naive self-distillation83.2046.5964.89−1.93+3.21
Self-distillation with teacher approximation error control91.36***47.98*69.67***+2.85+7.99
Training Method (Qwen3-8B)Safety (%) ↑Reasoning (%) ↑Overall (%) ↑Δ vs SFT (pp)Δ vs pretrained (pp)
Naive self-distillation91.8163.3177.56+0.38+1.71
Self-distillation with teacher approximation error control95.95***64.2880.11***+2.93+4.26

Significance level: * p < 0.05, *** p < 0.001 under two-sided permutation tests with 100.000 permutations

Comments on the paper

2 comments
AuthorsOpenReview ↗

Theory Extension: Connecting In-Context Approximation Errors to Constitutional On-Policy Self-Distillation

[TLDR] We have added Proposition 8, connecting our three-error decomposition to constitutional on-policy self-distillation. The teacher is the student itself given constitutional context, which approximates an unknown ideal aligned policy. The proposition shows that learning from this imperfect teacher still moves the student toward that ideal. We have tested the practical implications of this result and find that controlling all three in-context approximation errors during on-policy self-distillation improves overall performance by +4.78 / +2.55 percentage points over naive self-distillation and +2.85 / +2.93 points over SFT for Qwen3-1.7B / 8B, respectively.

Theoretical Result: Proposition 8

Fix a prompt and student-generated response prefix ss. Let πθ\pi_\theta and πθ+\pi_{\theta^+} denote the student before and after an update, πθ,R\pi_{\theta,R} the fixed contextual teacher, and πθ∗\pi_{\theta^*} the unknown ideal target. All distributions below are evaluated at ss. Our decomposition bounds the teacher's approximation error:

=\operatorname{osc}\log\frac{\pi_{\theta,R}}{\pi_{\theta^*}} \le \underbrace{\mathcal A(x,R)}_{\text{selection}} +\underbrace{\sum_{j\in R}\eta_j(x)}_{\text{interpretation}} +\underbrace{\iota(x,R)}_{\text{interaction}} =:B$$ Define the improvement in matching the teacher and the amount the student changes as $$\mathcal G_R(s) \=\mathrm{KL}(\pi_\theta\Vert\pi_{\theta,R}) -\mathrm{KL}(\pi_{\theta^+}\Vert\pi_{\theta,R}), \qquad \mathcal V(s)=\mathrm{TV}(\pi_{\theta^+},\pi_\theta).$$ The new proposition proves $$\mathrm{KL}(\pi_\theta\Vert\pi_{\theta^{\ast}})-\mathrm{KL}(\pi_{\theta^+}\Vert\pi_{\theta^{\ast}})\ge\mathcal G_R(s)-B\mathcal V(s).$$ Thus, the student improves whenever its gain from matching the teacher exceeds the teacher's error bound multiplied by how much the student changes. Crucially, reducing selection, interpretation, and interaction errors makes this condition easier to satisfy. ### Empirical Results: On-Policy Self-Distillation To investigate the practical implications of this result, we have conducted on-policy self-distillation experiments, training on SafeChain. We indeed find that under otherwise matched training and evaluation regimes, addressing these errors in the teacher's context during on-policy self-distillation improves the resulting student policies by +4.78 / +2.55 pp over naive self-distillation and +2.85 / +2.93 pp over the corresponding SFT baselines (1.7B / 8B), with trained students evaluated without context or privileged information on HarmBench, StrongREJECT, XSTest, MMLU-Pro, MATH-500, and GPQA-Diamond: **Qwen3-1.7B** | Method | Safety (%) ↑ | Reasoning (%) ↑ | Overall (%) ↑ | Δ vs SFT (pp) | Δ vs pretrained (pp) | | -------------------------------------------------------------- | --------------- | --------------- | --------------- | ------------- | -------------------- | | Naive self-distillation | 83.20 | 46.59 | 64.89 | −1.93 | +3.21 | | **Self-distillation with teacher approximation error control** | **91.36\*\*\*** | **47.98\*** | **69.67\*\*\*** | **+2.85** | **+7.99** | **Qwen3-8B** | Method | Safety (%) ↑ | Reasoning (%) ↑ | Overall (%) ↑ | Δ vs SFT (pp) | Δ vs pretrained (pp) | | -------------------------------------------------------------- | --------------- | --------------- | --------------- | ------------- | -------------------- | | Naive self-distillation | 91.81 | 63.31 | 77.56 | +0.38 | +1.71 | | **Self-distillation with teacher approximation error control** | **95.95\*\*\*** | **64.28** | **80.11\*\*\*** | **+2.93** | **+4.26** | Significance level: \* p < 0.05, \*\*\* p < 0.001 Further details about the experiments will be provided in a separate comment.
AuthorsOpenReview ↗

Experimental validation of the error decomposition

TL;DR: Reviewers 1STp, ma5y and fx3k asked whether the error terms (rule interpretation error η\eta, rule interaction error ι\iota, and clause selection error A\mathcal{A}) can be measured and whether the bound is informative. We added an experiment to answer this. The observed approximation error DRD_R remains below the bound in all tested settings, as the triangle inequality guarantees, and the bound becomes tighter as model performance improves. For Llama-3.1-70B, it lies only 0.3–0.6 log-odds units above the measured error, compared with a chance-level error of 2.94 in our setup.

Experimental setup

To make the individual error terms measurable, we construct deterministic logical riddles, similar in spirit to "Einstein logic puzzles", where a symbolic engine can determine the correct answer.

(a) When there is buhesal and redupum, you must sizis.

(b) When, and only when, another rule in this list requires sizis, and there is dehak, you must metru.

Situation. … there is buhesal; there is redupum; there is dehak.

Must you metru? Answer 1 or 0.

We read the model's log-odds of "1" against "0" and ask each case twice, once per selected rule alone and once with all selected rules together. In the example, neither rule alone requires metru (correct answer 0), but together they do (answer 1). For each prompt this gives the interpretation error η\eta (error on one rule alone), the interaction error ι\iota (extra error when the rules are combined) and the clause selection error A\mathcal{A} (error from selecting wrong rules, fixed by design). We can therefore compute all quantities in our theoretical upper bound:

DR≤A+∑jηj+ιD_R \leq \mathcal{A} + \sum_j \eta_j + \iota

Controlled experiments

1. Rule interpretation error

To study how accurately models interpret individual rules, we increase the number of conditions that must simultaneously hold for a rule to apply, from one to five. Each riddle in this setup contains only one rule, so the interaction and clause selection errors are zero by construction.

ModelConditions∑jηj\sum_j \eta_jDRD_RBoundGap
Llama-3.1-8B11.21.21.20
Llama-3.1-8B32.52.52.50
Llama-3.1-8B52.62.62.60
Qwen3.5-27B10.10.10.10
Qwen3.5-27B31.81.81.80
Qwen3.5-27B52.22.22.20
Llama-3.1-70B10.00.00.00
Llama-3.1-70B31.61.61.60
Llama-3.1-70B52.42.42.40

2. Rule interaction error

To isolate the effect of rule interactions, we construct chains of logical implications in which the conclusion of one rule activates the next. Reading each rule correctly is not enough: the model must combine them.

We increase the chain depth from 1 to 16 links while keeping the correct set of rules available, so that clause selection error remains zero.

ModelChain links∑jηj\sum_j \eta_jι\iotaDRD_RBoundGap
Llama-3.1-8B13.73.62.57.34.8
Llama-3.1-8B46.36.72.813.110.3
Llama-3.1-8B1614.415.82.930.227.3
Qwen3.5-27B10.52.01.92.50.6
Qwen3.5-27B40.92.62.63.51.0
Qwen3.5-27B163.53.12.96.63.7
Llama-3.1-70B10.21.71.72.00.3
Llama-3.1-70B40.32.62.63.00.4
Llama-3.1-70B160.53.02.93.50.5

3. Clause selection error

We vary the fraction of cases in the evaluation dataset for which a rule needed to correctly answer a question is missing (selection error).

ModelCases missing a rule∑jηj\sum_j \eta_jA\mathcal{A}ι\iotaDRD_RBoundGap
Llama-3.1-8B0%4.20.06.32.110.58.4
Llama-3.1-8B50%4.02.93.42.310.48.1
Llama-3.1-8B100%3.95.90.62.510.37.9
Qwen3.5-27B0%0.60.03.02.43.51.1
Qwen3.5-27B50%0.52.91.64.05.11.1
Qwen3.5-27B100%0.55.90.25.56.51.0
Llama-3.1-70B0%0.30.02.32.02.60.6
Llama-3.1-70B50%0.32.91.23.94.50.6
Llama-3.1-70B100%0.35.90.15.76.30.6

Overall findings:

  • The terms show where a model fails. Llama-3.1-70B reads each rule almost perfectly but fails to combine them (ι\iota rises).
  • The bound is tighter for capable models: the gap is 0.3 to 0.6 for Llama-3.1-70B, 0.6 to 3.7 for Qwen3.5-27B, and 4.8 to 27 for Llama-3.1-8B. Weaker models make errors on single rules that cancel when the rules are combined, so the bound grows while D_R does not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.