- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes HDPO for generating hierarchical software-evolution plans consisting of modules, implementation tasks, and file paths. HDPO modifies DPO by assigning larger weights to high-level module and task tokens. Training data are reconstructed from historical pull requests and commits, with synthetic rationales and hard-negative trajectories. Experiments on approximately 2,500 requests from one proprietary and three open-source repositories report substantial gains over SFT, weighted SFT, DPO, retrieval, and frontier-model baselines.
Critical strengths
1.The paper targets repository-level architectural planning rather than isolated code generation, and the module-task-file hierarchy provides a useful evaluation decomposition. 2.HDPO is evaluated across four distinct repositories with several relevant baselines, weight ablations, temporal splits, and human-audited LLM judgments. The reported gains over Step-DPO are large and consistent.
Critical weaknesses
1.The justification for the 4:2:1 weighting is insufficiently clear. Although the paper reports that 4:2:1 performs best among several configurations, it is unclear how this ratio was initially determined and whether it generalizes beyond the selected validation set and repositories. More systematic sensitivity analysis is needed. 2.The evaluation may overlook valid alternative plans. Module and file predictions are scored by exact match against historical patches, although multiple architectural plans may correctly implement the same feature. Human evaluation of non-matching predictions would strengthen the conclusions.
Questions for authors
- How was the 4:2:1 weighting ratio determined? Was it motivated by measured layer-wise gradients or selected through validation-set search?
- How sensitive is HDPO to nearby weight configurations, and does 4:2:1 remain optimal across different repositories? 3.How many exact-match failures correspond to valid alternative module or file plans according to human experts? 4.Does Weighted Impact correlate with actual implementation effort or development-time reduction, rather than only historical resolution duration?