- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EDIT, an energy-based framework for aggregating multiple reward models (RMs) into prompt-specific preference graphs for LLM alignment. The method combines continuous reward margins and candidate embeddings through a graph encoder and an amortized structure predictor. A generalized structural equation model provides a tractable graph-conditioned density, while a residual energy-based model captures dependencies not explained by the structural model. The framework is trained through SEM likelihood, noise-contrastive estimation, and adaptive negative sampling with a conditional flow. Acyclicity regularization and edge pruning produce a DAG that supports response ranking and preference-pair construction for downstream policy optimization.
The authors evaluate EDIT on RMB for response selection and AlpacaFarm for Best-of-N selection, DPO, and PPO. The method outperforms the reported baselines across the main evaluation metrics. Additional experiments examine reward overoptimization, correlated reward-model errors, structural recovery, and component ablations. The results suggest that combining structural and energy-based modeling can improve reward-model aggregation and robustness under optimization pressure.
Critical strengths
-
Promising integration of structural and energy-based preference modeling. EDIT combines reward margins, candidate representations, generalized SEMs, and residual energy modeling to capture dependencies beyond conventional scalar ensemble aggregation. This is an interesting approach to handling disagreement and correlated errors among reward models.
-
Consistent empirical improvements with meaningful ablations. EDIT outperforms the reported baselines on RMB and AlpacaFarm. In Table 1, its RMB average accuracy improves from 46.4% (PGED) to 49.5%, while PPO gold win rate increases from 71.9% to 75.2% with lower corresponding KL. Table 2 further demonstrates that removing the structural likelihood, energy correction, or adaptive flow reduces performance. The robustness experiments under correlated corruption also strengthen the empirical contribution.
Critical weaknesses
-
The connection between density modeling and preference correctness needs stronger justification. The core training objective (Eqs. 11–16) optimizes structural likelihood, noise-contrastive estimation, and graph regularization, without directly supervising the learned preference ordering. Proposition 2 establishes conditional density consistency under fixed structural components, but does not guarantee that the learned DAG recovers the correct preference directions. In particular, a flexible residual energy model could potentially explain similar data distributions under different graph structures. Although the synthetic structure recovery and ranking experiments provide useful empirical evidence, the paper does not fully explain why density-based structure learning should systematically correct shared RM preference errors. This limits the strength of the methodological interpretation.
-
Important experimental controls and ensemble configurations remain unclear. Section 5.1 specifies five Pythia-70M RMs for AlpacaFarm, but describes the RMB evaluation using an 8B Skywork RM without clearly explaining how the multiple RM views required by EDIT are constructed. Moreover, EDIT uses reward embeddings and additional training on 10K prompts, while it is unclear whether the strongest aggregation baselines have access to equivalent information and training resources. These details are important for determining how much of the reported improvement is attributable to the proposed SEM–EBM formulation rather than richer input representations or additional model capacity.
Questions for authors
-
Preference identification: What specifically encourages the learned graph edges to represent correct preference directions rather than merely statistical dependencies among candidate representations? How should Proposition 2 be interpreted in relation to preference ranking consistency?
-
RMB ensemble configuration: How many RMs are used to construct the RMB preference graphs, and which checkpoints provide the different views? The implementation section mentions an 8B Skywork RM, whereas EDIT requires multiple RM scores and embeddings. Is EDIT trained separately for RMB and AlpacaFarm?
-
Controlled comparisons: Do PGED and the other aggregation baselines use exactly the same underlying RM outputs and candidate representations as EDIT? Are additional training data and computational resources comparable? Clarifying these conditions would help determine the independent contribution of the structural and energy-based components.