acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Deployment-Aware Alignment for Large Language Models

Abstract

Large language model (LLM) alignment aims to ensure that models behave consistently with human intentions, preferences, and values while reducing harmful or undesirable outputs. Existing post-training alignment methods typically assume a static deployment setting, where each input directly yields a single model output. However, the final alignment performance of deployed LLMs is also shaped by prompt engineering and inference-time algorithms, which form a dynamic deployment environment that prior alignment methods largely overlook. To address this mismatch between static alignment and dynamic deployment, we propose DynaAlign, a dynamic deployment-aware alignment framework that jointly optimizes the LLM policy and the prompt enhancer under a given inference-time algorithm. DynaAlign formulates alignment as a KL-regularized end-to-end win-rate maximization problem and solves it with an alternating optimization procedure, where the prompt enhancer is updated using downstream utility reward and the LLM policy is trained with an inference-aware reward tailored to the target inference-time algorithm. Extensive experiments across multiple benchmarks show that DynaAlign consistently improves end-to-end performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, alignment, dynamic deployment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h2eXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the mismatch between conventional LLM alignment objectives and real-world dynamic deployment settings. It proposes DynaAlign, a framework that jointly models the Prompt Enhancer, LLM Policy, and inference-time algorithm as an end-to-end alignment system. The authors formulate a KL-regularized win-rate maximization objective, derive a block-wise optimization approach for the Prompt Enhancer and LLM Policy, and train them using alternating PPO. Experiments cover two model families, multiple helpfulness and safety benchmarks, and both Best-of-NN and Worst-of-NN inference algorithms. The results show consistent improvements across the reported settings, while ablation studies support the importance of downstream utility rewards, inference-aware rewards, and alternating optimization.

Critical strengths

1. The research problem is practically relevant and offers a system-level perspective on alignment optimization. Existing methods typically focus on optimizing the LLM policy alone, overlooking how the prompt enhancer and inference algorithm jointly affect the distribution of final responses. This paper brings these components together under a unified optimization objective, with the goal of improving end-to-end performance in realistic deployment settings. I find this problem formulation practically valuable, and it is one of the main reasons I support accepting this paper.

2. The experimental results are consistent, and the ablation studies support the key design choices. The paper reports consistent improvements across two model families, multiple benchmarks, and two types of inference algorithms. The authors also provide ablation studies on three key components, analyses under different sampling budgets, and human evaluations. In particular, the full method performs better than the variant without alternating training, which supports the argument that these components benefit from being optimized together.

Critical weaknesses

1. The connection between the theoretical objective and the actual training procedure is not sufficiently established.

Proposition 2 depends on the derivative of the inference operator and the unknown optimal policy, while the implementation uses an InfAlign-inspired calibrated reward transformation. It remains unclear whether this practical approximation preserves the intended optimization direction or whether alternating PPO improves the unified objective. This gap weakens the paper's central claim of coordinated end-to-end optimization.

2. The evaluation of alignment quality and practical generalizability remains limited.

Human evaluation covers only one benchmark with two annotators. The prompt enhancer may improve evaluator scores by changing response length, scope, or wording without necessarily preserving user intent. Moreover, experiments mainly focus on N=4N=4 with Best-of-NN and Worst-of-NN, while training costs are about 25% higher than InfAlign. More cost-matched comparisons and evaluations of semantic fidelity and safety would strengthen the paper.

Questions for authors
  1. Is there a provable connection between the practical inference-aware reward transformation and the optimal reward derived in Proposition 2? Could the authors report how the unified objective J(E,π)J(E,\pi) changes during alternating training to verify that the proposed method actually optimizes it?

  2. Could the authors compare DynaAlign with a combination of independently optimized Prompt Enhancer and InfAlign under the same training compute budget? How much of the improvement comes from joint optimization rather than additional computation?

  3. Have the authors evaluated semantic consistency before and after prompt rewriting, as well as the actual harmful-response rate under Worst-of-NN? These results would help determine whether the improved win rates reflect genuine gains in user alignment and safety.

Reviewer_xX3TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM alignment when the deployed system contains not only an LLM, but also a prompt enhancer and an inference-time procedure. The proposed DynaAlign framework optimizes the prompt enhancer and the LLM policy alternately under an end-to-end win-rate objective. The enhancer is trained using the quality of the final response produced by the downstream pipeline, while the LLM is updated with a transformed reward intended to reflect Best-of-N or Worst-of-N inference. Experiments with Qwen and Llama models show consistent improvements over standard and inference-aware alignment baselines on helpfulness and safety benchmarks.

Critical strengths
  • The motivation is clear and practically relevant. When prompt rewriting and inference-time selection are part of the deployed system, optimizing only the standalone LLM may be suboptimal.
  • Using the final downstream outcome to train the prompt enhancer is sensible, and alternating the enhancer and policy updates provides a coherent way to optimize the whole pipeline.
  • The experiments are comprehensive, and the improvements are consistent across the tested model families and inference settings.
Critical weaknesses
  • The connection between the policy-side theory and the practical algorithm could be explained more clearly. Proposition 2 characterizes a reward that depends on the unknown optimal policy, while the implementation uses the calibrate-and-transform reward from InfAlign. It is unclear whether this practical reward approximates the derived reward in the joint setting or should instead be viewed as a reasonable surrogate.
  • The cross-checkpoint analysis is not fully conclusive, since later checkpoints tend to perform better with many counterparts, not only with their matched checkpoint.
  • Since the method actively rewrites user inputs, it would be useful to verify that the gains do not come at the cost of changing the intended scope of the request. Some qualitative examples favor broader or more detailed prompts, but the paper does not separately measure semantic preservation or control for response length.
Questions for authors

The title describes the setting as “dynamic deployment,” although the prompt enhancer and inference-time procedure are fixed for a given training run. Could the authors clarify what “dynamic” means here?

Reviewer_GJjyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DynaAlign, which jointly trains a prompt enhancer EE and an LLM policy for a fixed inference-time procedure TT. It formulates a KL-regularized objective for the win rate of the complete deployment pipeline against a reference pipeline. In practice, the enhancer is trained with a Monte Carlo estimate of downstream utility, while the policy is trained using InfAlign-style calibrated and transformed rewards, with the two updates alternated through PPO. Experiments on Qwen3-4B and Llama-3.1-8B cover four helpfulness benchmarks with Best-of-NN decoding and two safety benchmarks with Worst-of-NN decoding. The reported gains are consistent across a DeepSeek judge and a reward-model evaluator, with additional support from a small human evaluation.

Critical strengths
  1. The deployment-level objective is practically relevant. Prompt rewriting and sample selection both affect the response a user receives. Treating these components as part of the alignment target is a useful extension of policy-only inference-aware alignment.

  2. The empirical evaluation provides several complementary checks. The paper evaluates two model families and both helpfulness and safety settings. Tables 1–2 use an independent LLM judge, Appendix C includes human comparisons, and the DynaAlign-NA ablation directly examines sequential versus alternating training. These are meaningful pieces of evidence for the proposed approach.

Critical weaknesses
  1. The connection between the practical policy reward and the stated objective remains incomplete. Proposition 2 gives an implicit optimality condition involving the derivative of TT at the unknown optimal policy. Section 3.3 acknowledges that this reward is intractable, but Eqs. (12) and (16)–(18) then replace it with an InfAlign-style transformation calibrated against the reference pipeline without inference-time selection. The paper does not establish when this surrogate approximates the objective-derived reward, or when its policy update improves Eq. (5). This matters because the method is repeatedly described as optimizing the two blocks of one unified objective. The enhancer update has a more direct connection to that objective than the implemented policy update does.

  2. The evidence isolating alternating coordination needs a clearer budget and uncertainty account. DynaAlign-NA is the appropriate two-stage control and is already included. However, its enhancer and policy update counts, rollout cost, and total compute relative to DynaAlign are not specified clearly enough to interpret the comparison. Appendix C reports approximately 25% more training time than InfAlign, and the key ablations are evaluated with the Skywork evaluator rather than the independent judge. Since several main-table gains are only a few percentage points and Vicuna Eval contains 80 queries, uncertainty estimates and a budget-accounted comparison would make the attribution substantially stronger.

  3. The safety results do not yet distinguish reduced harmfulness from broader changes in responsiveness. Figures 8–9 deliberately redirect harmful requests toward safe topics, which can be an appropriate safety behavior. The missing evidence is whether the same mechanism preserves benign requests, including safety-related but legitimate ones, and whether Worst-of-NN gains correspond to lower absolute unsafe-response rates. Relative preference wins and the general-capability tests in Appendix C do not directly measure inappropriate redirection or over-refusal. The evaluation description should also explicitly confirm that judges receive the original user query.

Questions for authors
  1. Under what assumptions do the implemented policy rewards in Eqs. (12) and (16)–(18) approximate the policy-block reward for Eq. (5)? If they are an empirical surrogate, could you state that distinction explicitly and clarify the corresponding optimization claim?

  2. Are DynaAlign and DynaAlign-NA matched in enhancer/policy updates and rollout cost? Please report these budgets and uncertainty for their comparison, ideally with the independent judge, and account for the additional training cost relative to InfAlign.

  3. Do all evaluators see the original query? Please clarify the safety-specific judging and Worst-of-NN selection criterion, and provide a focused check of absolute unsafe-response rates together with benign-query fidelity or over-refusal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.