acceptodds
Under review as a conference paper at ICLR 2027

Energy-based DAG Structure Refinement for Large Language Model Preference Alignment

Abstract

Aligning large language models (LLMs) with human preferences remains a central challenge in modern AI. Reward models (RMs) serve as proxies for human judgment but are highly susceptible to overoptimization and reward hacking. RM ensembles provide complementary evaluations, but scalar pooling compresses their disagreements and graph aggregation can retain conflicting edges. In this paper, we propose **E**nergy-based **D**AG s**T**ructure ref**I**nement (EDIT) to learn a prompt-specific preference graph from multiple RMs. A structural equation model (SEM) supplies a tractable graph-conditioned density, and a residual energy-based model (EBM) fits joint patterns beyond this base model. We train the structural module and energy function with SEM likelihood and noise contrastive estimation (NCE) using adaptive negatives from a conditional flow. Acyclicity regularization and edge pruning yield a directed acyclic graph (DAG) for consistent ranking. Extensive experiments on RMB and AlpacaFarm demonstrate state-of-the-art performance in both response selection and policy optimization.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
DAG Structure Refinement, Energy-based Model, LLM Preference Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_55xLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EDIT, an energy-based framework for aggregating multiple reward models (RMs) into prompt-specific preference graphs for LLM alignment. The method combines continuous reward margins and candidate embeddings through a graph encoder and an amortized structure predictor. A generalized structural equation model provides a tractable graph-conditioned density, while a residual energy-based model captures dependencies not explained by the structural model. The framework is trained through SEM likelihood, noise-contrastive estimation, and adaptive negative sampling with a conditional flow. Acyclicity regularization and edge pruning produce a DAG that supports response ranking and preference-pair construction for downstream policy optimization.

The authors evaluate EDIT on RMB for response selection and AlpacaFarm for Best-of-N selection, DPO, and PPO. The method outperforms the reported baselines across the main evaluation metrics. Additional experiments examine reward overoptimization, correlated reward-model errors, structural recovery, and component ablations. The results suggest that combining structural and energy-based modeling can improve reward-model aggregation and robustness under optimization pressure.

Critical strengths
  1. Promising integration of structural and energy-based preference modeling. EDIT combines reward margins, candidate representations, generalized SEMs, and residual energy modeling to capture dependencies beyond conventional scalar ensemble aggregation. This is an interesting approach to handling disagreement and correlated errors among reward models.

  2. Consistent empirical improvements with meaningful ablations. EDIT outperforms the reported baselines on RMB and AlpacaFarm. In Table 1, its RMB average accuracy improves from 46.4% (PGED) to 49.5%, while PPO gold win rate increases from 71.9% to 75.2% with lower corresponding KL. Table 2 further demonstrates that removing the structural likelihood, energy correction, or adaptive flow reduces performance. The robustness experiments under correlated corruption also strengthen the empirical contribution.

Critical weaknesses
  1. The connection between density modeling and preference correctness needs stronger justification. The core training objective (Eqs. 11–16) optimizes structural likelihood, noise-contrastive estimation, and graph regularization, without directly supervising the learned preference ordering. Proposition 2 establishes conditional density consistency under fixed structural components, but does not guarantee that the learned DAG recovers the correct preference directions. In particular, a flexible residual energy model could potentially explain similar data distributions under different graph structures. Although the synthetic structure recovery and ranking experiments provide useful empirical evidence, the paper does not fully explain why density-based structure learning should systematically correct shared RM preference errors. This limits the strength of the methodological interpretation.

  2. Important experimental controls and ensemble configurations remain unclear. Section 5.1 specifies five Pythia-70M RMs for AlpacaFarm, but describes the RMB evaluation using an 8B Skywork RM without clearly explaining how the multiple RM views required by EDIT are constructed. Moreover, EDIT uses reward embeddings and additional training on 10K prompts, while it is unclear whether the strongest aggregation baselines have access to equivalent information and training resources. These details are important for determining how much of the reported improvement is attributable to the proposed SEM–EBM formulation rather than richer input representations or additional model capacity.

Questions for authors
  1. Preference identification: What specifically encourages the learned graph edges to represent correct preference directions rather than merely statistical dependencies among candidate representations? How should Proposition 2 be interpreted in relation to preference ranking consistency?

  2. RMB ensemble configuration: How many RMs are used to construct the RMB preference graphs, and which checkpoints provide the different views? The implementation section mentions an 8B Skywork RM, whereas EDIT requires multiple RM scores and embeddings. Is EDIT trained separately for RMB and AlpacaFarm?

  3. Controlled comparisons: Do PGED and the other aggregation baselines use exactly the same underlying RM outputs and candidate representations as EDIT? Are additional training data and computational resources comparable? Clarifying these conditions would help determine the independent contribution of the structural and energy-based components.

Reviewer_hR8qOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a novel framework EDIT (Energy-based DAG Structure Refinement) which is used to combine multiple reward models for preference alignment. The authors argue that simple aggregation of RM ensembles loses information related to disagreements and graph aggregation provides a way to retain the disagreements. The authors use RM scores and hidden embeddings to learn a preference weighted graph. The training process combines a structural equation model (SEM) with energy based correction, noise contrastive estimation and an adaptive flow sampler. Empirical studies show that this framework improves the performance over other reward-model ensemble and graph-aggregation baselines on two datasets. The ablation studies show the importance of individual components of the framework and also show that this framework is more robust to correlated RM errors.

Critical strengths
  1. The paper addresses an important problem. Preference alignment is susceptible to reward hacking and the use of RM ensembles is an established paradigm to mitigate reward hacking. However, RMs can disagree and this framework to capture RM disagreements. The empirical studies show an improvement over other graph--aggregation based frameworks, simple scalar aggregation and the ablations provide evidence that each individual component provides contributions to the framework.
  2. The experiments show that the framework is more robust to noisy RM labels in comparison to the baselines.
Critical weaknesses
  1. This framework introduces substantial complexity through SEM-based density modeling, a residual energy correction and an adaptive flow sampler. While the ablation studies show the contributions of each component in this framework, the paper lacks comparisons against a simpler GNN and pair-scoring architecture with a simpler training process. Perhaps a simple baseline could use the same GNN, pair-scoring network but train using pairwise-cross entropy loss using soft preferences derived from ensemble margins. This would help establish whether the additional complexity is necessary for the reported improvements.
  2. The authors do not quantify and address the additional computational complexity introduced in the framework. The training process needs SEM density evaluation, energy-based contrastive learning with multiple negative samples, alternating updates to adaptive flow sampler. The inference also needs GNN encoding, pariwise edge prediction and topological sort. It is not clear whether the reported improvements justify the additional computational cost.
Questions for authors

Please see both the weaknesses: Would it be possible to compare the framework with similar GNN and pairwise scoring architecture using a simpler training process?Additionally could you add information on the additional computational cost introduced by this framework relative to the baselines?

Reviewer_hyYxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EDIT (Energy-based DAG sTructure refInement), which aggregates judgments from multiple reward models (RMs) to rank candidate responses. EDIT learns relations between responses from RM reward margins and embeddings. It removes cycles to form a directed acyclic graph (DAG). EDIT uses the resulting ranking for response selection and uses preference pairs constructed from the ranking for policy optimization with DPO and PPO. This paper reports better results than the baselines on RMB and AlpacaFarm and also tests robustness to errors shared by multiple RMs.

Critical strengths

This paper proposes a framework that learns relations between candidate responses as a DAG from evaluations by multiple RMs. It uses the resulting ranking for DPO and PPO training. The design treats RMs as imperfect proxies, handles disagreements between the models, and uses the results for existing policy optimization methods. This design is reasonable.

Critical weaknesses
  • W1 (Unclear settings for the main comparisons): For the mean and worst-case RM aggregation methods in Table 1, the number of candidate responses and the way training pairs are constructed for DPO and PPO are unclear. I cannot confirm whether the settings match those of EDIT, so it is difficult to assess whether the reported gains come from the proposed method.

  • W2 (Effects of the DAG and embeddings): The paper includes ablation studies. However, comparisons with methods that rank responses from the same inputs without a DAG, or with methods that do not use RM embeddings, are unclear. Therefore, the benefits and effects of these designs seem unclear.

Questions for authors
  • Q1: Could you explain how the DPO and PPO training procedures for the mean and worst-case RM aggregation methods in Table 1 compare with EDIT? In particular, I would like to check the number of candidate responses, pair selection, and training budgets. I would like to check whether differences in these settings explain the gains over simple aggregation.

  • Q2: For example, do you have comparison results with a method that ranks responses from the same inputs without a DAG, or with a method that excludes RM embeddings? I would like to check how much the DAG and embeddings each contribute to the gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.