- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the mismatch between conventional LLM alignment objectives and real-world dynamic deployment settings. It proposes DynaAlign, a framework that jointly models the Prompt Enhancer, LLM Policy, and inference-time algorithm as an end-to-end alignment system. The authors formulate a KL-regularized win-rate maximization objective, derive a block-wise optimization approach for the Prompt Enhancer and LLM Policy, and train them using alternating PPO. Experiments cover two model families, multiple helpfulness and safety benchmarks, and both Best-of- and Worst-of- inference algorithms. The results show consistent improvements across the reported settings, while ablation studies support the importance of downstream utility rewards, inference-aware rewards, and alternating optimization.
Critical strengths
1. The research problem is practically relevant and offers a system-level perspective on alignment optimization. Existing methods typically focus on optimizing the LLM policy alone, overlooking how the prompt enhancer and inference algorithm jointly affect the distribution of final responses. This paper brings these components together under a unified optimization objective, with the goal of improving end-to-end performance in realistic deployment settings. I find this problem formulation practically valuable, and it is one of the main reasons I support accepting this paper.
2. The experimental results are consistent, and the ablation studies support the key design choices. The paper reports consistent improvements across two model families, multiple benchmarks, and two types of inference algorithms. The authors also provide ablation studies on three key components, analyses under different sampling budgets, and human evaluations. In particular, the full method performs better than the variant without alternating training, which supports the argument that these components benefit from being optimized together.
Critical weaknesses
1. The connection between the theoretical objective and the actual training procedure is not sufficiently established.
Proposition 2 depends on the derivative of the inference operator and the unknown optimal policy, while the implementation uses an InfAlign-inspired calibrated reward transformation. It remains unclear whether this practical approximation preserves the intended optimization direction or whether alternating PPO improves the unified objective. This gap weakens the paper's central claim of coordinated end-to-end optimization.
2. The evaluation of alignment quality and practical generalizability remains limited.
Human evaluation covers only one benchmark with two annotators. The prompt enhancer may improve evaluator scores by changing response length, scope, or wording without necessarily preserving user intent. Moreover, experiments mainly focus on with Best-of- and Worst-of-, while training costs are about 25% higher than InfAlign. More cost-matched comparisons and evaluations of semantic fidelity and safety would strengthen the paper.
Questions for authors
-
Is there a provable connection between the practical inference-aware reward transformation and the optimal reward derived in Proposition 2? Could the authors report how the unified objective changes during alternating training to verify that the proposed method actually optimizes it?
-
Could the authors compare DynaAlign with a combination of independently optimized Prompt Enhancer and InfAlign under the same training compute budget? How much of the improvement comes from joint optimization rather than additional computation?
-
Have the authors evaluated semantic consistency before and after prompt rewriting, as well as the actual harmful-response rate under Worst-of-? These results would help determine whether the improved win rates reflect genuine gains in user alignment and safety.