acceptodds
Under review as a conference paper at ICLR 2027

HDPO: Aligning Large Language Models to Software Architecture via Hierarchical Preference Optimization

Abstract

Large Language Models struggle with software evolution in complex repositories, often taking a “Semantic Shortcut” that ignores architectural hierarchy. We introduce Hierarchical Direct Preference Optimization (HDPO), a novel alignment framework that enforces structural consistency by decomposing feature generation into a rigorous three-tier mapping of Modules, Tasks, and Files. Across major open-source ecosystems (e.g., Django, Kafka, Kubernetes), HDPO achieves an average structural accuracy of 40.3%, drastically outperforming Step-SFT (14.7%), Step-Weighted-SFT (16.5%), and Step-DPO (19.0%). Similarly, regarding Average Weighted Impact across these ecosystems, HDPO (51.0%) decisively outperforms Step-DPO (25.0%), Step-Weighted-SFT (22.3%), and Step-SFT (20.7%).

Primary area:
reinforcement learning
Keywords:
Hierarchical Planning, Direct Preference Optimization (DPO), LLM Alignment, Software Evolution, Reinforcement Learning, Code Generation, Structural Hallucination, Developer Productivity, Industrial Software Engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7a14OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes HDPO for generating hierarchical software-evolution plans consisting of modules, implementation tasks, and file paths. HDPO modifies DPO by assigning larger weights to high-level module and task tokens. Training data are reconstructed from historical pull requests and commits, with synthetic rationales and hard-negative trajectories. Experiments on approximately 2,500 requests from one proprietary and three open-source repositories report substantial gains over SFT, weighted SFT, DPO, retrieval, and frontier-model baselines.

Critical strengths

1.The paper targets repository-level architectural planning rather than isolated code generation, and the module-task-file hierarchy provides a useful evaluation decomposition. 2.HDPO is evaluated across four distinct repositories with several relevant baselines, weight ablations, temporal splits, and human-audited LLM judgments. The reported gains over Step-DPO are large and consistent.

Critical weaknesses

1.The justification for the 4:2:1 weighting is insufficiently clear. Although the paper reports that 4:2:1 performs best among several configurations, it is unclear how this ratio was initially determined and whether it generalizes beyond the selected validation set and repositories. More systematic sensitivity analysis is needed. 2.The evaluation may overlook valid alternative plans. Module and file predictions are scored by exact match against historical patches, although multiple architectural plans may correctly implement the same feature. Human evaluation of non-matching predictions would strengthen the conclusions.

Questions for authors
  1. How was the 4:2:1 weighting ratio determined? Was it motivated by measured layer-wise gradients or selected through validation-set search?
  2. How sensitive is HDPO to nearby weight configurations, and does 4:2:1 remain optimal across different repositories? 3.How many exact-match failures correspond to valid alternative module or file plans according to human experts? 4.Does Weighted Impact correlate with actual implementation effort or development-time reduction, rather than only historical resolution duration?
Reviewer_N17nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper “HDPO: Aligning Large Language Models to Software Architecture via Hierarchical Preference Optimization” presents a variant of DPO that introduces token weights, based on architectural relationships. Tokens that describe the relationship on module levels have a weight of 4, on the task level a weight of 2, and on the file level a weight of 1. Based on a dataset that is collected for one proprietary system and three open-source systems, the authors align a Mistral-7B model using this approach and compare this with standard DPO, SFT, and SFT with the same weighting for the CE loss. The results show that HDPO yields the best performance.

Critical strengths
  1. The approach seems to work well, though this was only demonstrated with a single, relatively small, LLM. Still, the gap to the other tuning approaches shows that this worked well in this setting.
  2. The data set creation and overall descriptions of the method are very good and it is easy to follow how the model is trained.
Critical weaknesses
  1. I am honestly not sure what is being predicted, based on the description in the paper. What is the input into the post-trained LLM and how is the expected output defined? My guess is that the output are modules and files related to some problem. But maybe it is also tests that should be executed. I guess the root cause of this problem is that I am not sure what the “experiments” are. This makes the actual contribution and also rigor of the experiments hard to judge for me.
  2. The novelty of weighted DPO approaches is unclear. Since I only broadly follow the post-training literature, I could not name a specific approach prior to this review. But since using a weighted loss function is something fairly common and straightforward when dealing with biases between different aspects in the evaluation, I considered it likely that there already is work on this. With a shallow 2-minute search, I already found three papers that all seem to induce weights on tokens using different approaches:
Questions for authors
  1. What exactly is the task, i.e., what are the inputs, the expected outputs, and what is the exact role of the LLM-as-a-judge approach that is only mentioned in the appendix?
  2. How does this work compare to other weighting modifications of post-training methods, ideally not only DPO, but also weighted RLHF and GRPO.
Reviewer_39ZmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets cross-module feature planning in large repositories, arguing that the bottleneck is scoping what to change across modules, not code quality. Given a feature request, the model outputs three layers: affected modules, an ordered task plan, and file paths. The proposed HDPO weights DPO token log-ratios by layer, 4:2:1 for modules, tasks, and files, and adds a complexity curriculum. Training data come from merged PRs in a proprietary Ads monorepo plus Django, Kafka, and Kubernetes. On Mistral-7B, HDPO reports 40.3% end-to-end accuracy on the open-source repositories versus 19.0% for Step-DPO, and higher duration-weighted Weighted Impact.

Critical strengths
  1. A real and underexplored problem. Code LLMs mostly optimize local correctness. Scoping changes across modules in large repositories is a practical bottleneck that current benchmarks ignore.
  2. Realistic data source. Ground truth comes from merged pull requests in real repositories, with a temporal split to limit leakage.
Critical weaknesses
  1. Limited novelty. The method just multiplies DPO token log-ratios by fixed layer weights. Similar weighted or hierarchical DPO methods, such as HiPO, CHiP, and HPL, are not cited. The Step-DPO baseline is mislabeled as unweighted DPO.
  2. Implausible results. Nearly all three-run means are integers or end in .5. Within each domain, Gemini-3.1 equals Gemini-2.5 plus a near-constant offset. Raw per-run outputs are needed.
  3. Outdated backbone and baselines. Only Mistral-7B-v0.2 from 2023 is used. There is no comparison with modern models such as Qwen3.5 or with code agents such as SWE-agent.
Questions for authors
  1. Verifiability. Please release per-run predictions and the open-source splits, and explain why three-run means are almost all integers and why the two Gemini models differ by a near-constant offset.
  2. Stronger comparisons. Please compare against segment-weighted DPO such as HiPO, and report results with a modern backbone such as Qwen3.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.