- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes POP, a method for constructing preference training data for open-ended tasks from unlabeled text. A single model handles data generation and scoring. It first generates a question and a reference answer from a source document, then generates multiple candidate answers given only the question. It constructs a rubric using the source document, reference answer, and candidate responses. During grading, the model receives only the question, rubric, and current candidate answer. The highest- and lowest-scoring answers form a preference pair for DPO training. This process requires neither additional human preference annotations nor a stronger external teacher. The authors report improvements over the initial models on long-form healthcare QA, creative writing, and instruction following.
Critical strengths
-
Constructing training data for open-ended tasks from raw text has practical value. POP produces DPO data without additional labeled questions, expert-written rubrics, or preference labels from a stronger model. SPICE obtains rewards from verifiable answers, whereas POP uses document-grounded rubrics to evaluate long-form answers that are difficult to assess through simple correctness checks. This is a useful extension: information from the source text informs the evaluation criteria, and differences in response quality provide the training signal.
-
The evaluation covers multiple tasks and different initial models. In Table 2, the base model improves from 26.71 to 29.64 on HealthBench500. The instruction-tuned model improves from 59.82 to 62.64 on Creative Writing V3 and from 70.30 to 72.34 on IFEval strict-prompt accuracy. IFEval uses rule-based scoring, providing evidence beyond LLM judging. Appendix G also reports healthcare experiments with LLaMA3.1-8B-Instruct. However, on IFEval with the base model, SFT on reference answers scores 37.46, above POP's 36.72, so POP does not outperform every baseline.
-
The paper is well organized, and the method is easy to follow.
Critical weaknesses
-
W1. The methodological novelty is incremental. The Self-Rewarding Language Models framework already uses self-scoring and DPO with the highest- and lowest-scoring responses; RLCF uses candidate-driven weighted checklists; and SPICE uses document-based question generation and information asymmetry between roles. Their supervision and verification settings differ: SRLM uses human-provided seed data and a fixed question generator, RLCF uses stronger external models, and SPICE targets verifiable answers. POP makes a meaningful distinction by handling open-ended long-form responses without additional human-written rubrics or a stronger teacher. Nevertheless, its novelty lies mainly in integrating existing mechanisms and extending their application; the individual components offer limited novelty.
-
W2. Comparisons with other post-training methods for open-ended tasks remain limited. The paper includes CPT, SFT, and internal ablations, but comparisons with the closest self-rewarding and rubric-based RL approaches are mainly qualitative. Differences in supervision constrain direct comparisons, leaving the current results insufficient to fully establish POP's relative advantages over these approaches.
-
W3. Ablation evidence for the role of pretraining text is limited to healthcare. In Figure 5, POP and Eval w/o D have overall scores of approximately 30 and 29, respectively. Directly providing the source document during rubric generation yields a modest overall gain, with mixed results across scoring dimensions. Removing documents from the entire pipeline (w/o D) reduces the overall score to about 27, supporting their value to the full pipeline. However, these ablations cover only HealthBench500. Corresponding experiments on creative writing and instruction following are not reported, leaving it unclear whether this particular design provides similar benefits on those tasks.
Questions for authors
See Weaknesses.
2 comments
Dear Reviewer,
With the author response window closing soon, we would deeply appreciate it if you could go through our responses and reconsider your rating if they address your concerns.
Thank you again for your time and effort in reviewing our paper.
Best regards,
The Paper Authors.
Although a fair comparison is difficult, we compare POP against the closest counterpart, SAO [1]. It synthesizes questions and responses and ranks the responses, all with the same LLM. The question-response pairs are used to train the LLM via preference optimization, similar to ours. The major difference between SAO and POP is that SAO uses strong LLM supervision for corpus curation. SAO creates questions from Persona Hub [2]. Persona Hub is a heavily compressed and reformatted version of raw web text, where the compression and reformatting are done by strong LLMs such as GPT-4.
We adapted SAO’s data synthesis and training pipeline to the three task settings that we use. We also controlled the final training set size to be similar to ours (SAO synthesizes 2935/1119/2935 preference pairs for healthcare/creative writing/instruction following, respectively). We show the results on Qwen-2.5-7B-Base as follows.
Despite using fewer supervisions (SAO uses stronger LLM supervisions for corpus curation, but POP does not), POP outperforms SAO on creative writing and instruction following tasks. On healthcare QA, POP slightly underperforms SAO by 0.74%.
[1] Yin et al. “Aligning Large Language Models via Fully Self-Synthetic Data”. ACL, 2026. [2] Ge et al. “Scaling Synthetic Data Creation with 1,000,000,000 Personas”. arXiv preprint arXiv:2406.20094.