acceptodds
Under review as a conference paper at ICLR 2027

Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text

Abstract

Self-play has recently emerged as a promising paradigm for post-training Large Language Models (LLMs). In self-play, the target LLM creates the task input (e.g., a question), which it then addresses itself by producing a task output (e.g., an answer). A reward model evaluates the output, and the rewards are used to train the LLM, typically via Reinforcement Learning (RL). A key benefit of self-play for post-training LLMs is its minimal supervision costs: self-play avoids the need for high-quality input-output pairs traditionally constructed by humans or expensive proprietary models. Existing work, however, explores self-play only for verifiable tasks, such as math and coding, for which objective ground truth is available and easily checkable. In this paper, we seek to extend self-play to more realistic open-ended tasks. We propose \ourmethod, the first fully autonomous LLM self-play framework on open-ended tasks. It uses the same LLM to synthesize evaluation rubrics along with each input-output pair. The rubric is used to evaluate outputs and train the model. Crucially, we ground the framework on a content-rich pretraining corpus to (1) enable an exploitable generation-verification gap and reduce reward hacking, and (2) prevent mode collapse. On Qwen-2.5-7B, \ourmethod increases the performance of both the pretrained base model and instruction-tuned model on multiple tasks ranging from long-form healthcare QA to creative writing and instruction following.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, self-play, open domains

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ibXiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes POP, a method for constructing preference training data for open-ended tasks from unlabeled text. A single model handles data generation and scoring. It first generates a question and a reference answer from a source document, then generates multiple candidate answers given only the question. It constructs a rubric using the source document, reference answer, and candidate responses. During grading, the model receives only the question, rubric, and current candidate answer. The highest- and lowest-scoring answers form a preference pair for DPO training. This process requires neither additional human preference annotations nor a stronger external teacher. The authors report improvements over the initial models on long-form healthcare QA, creative writing, and instruction following.

Critical strengths
  • Constructing training data for open-ended tasks from raw text has practical value. POP produces DPO data without additional labeled questions, expert-written rubrics, or preference labels from a stronger model. SPICE obtains rewards from verifiable answers, whereas POP uses document-grounded rubrics to evaluate long-form answers that are difficult to assess through simple correctness checks. This is a useful extension: information from the source text informs the evaluation criteria, and differences in response quality provide the training signal.

  • The evaluation covers multiple tasks and different initial models. In Table 2, the base model improves from 26.71 to 29.64 on HealthBench500. The instruction-tuned model improves from 59.82 to 62.64 on Creative Writing V3 and from 70.30 to 72.34 on IFEval strict-prompt accuracy. IFEval uses rule-based scoring, providing evidence beyond LLM judging. Appendix G also reports healthcare experiments with LLaMA3.1-8B-Instruct. However, on IFEval with the base model, SFT on reference answers scores 37.46, above POP's 36.72, so POP does not outperform every baseline.

  • The paper is well organized, and the method is easy to follow.

Critical weaknesses
  • W1. The methodological novelty is incremental. The Self-Rewarding Language Models framework already uses self-scoring and DPO with the highest- and lowest-scoring responses; RLCF uses candidate-driven weighted checklists; and SPICE uses document-based question generation and information asymmetry between roles. Their supervision and verification settings differ: SRLM uses human-provided seed data and a fixed question generator, RLCF uses stronger external models, and SPICE targets verifiable answers. POP makes a meaningful distinction by handling open-ended long-form responses without additional human-written rubrics or a stronger teacher. Nevertheless, its novelty lies mainly in integrating existing mechanisms and extending their application; the individual components offer limited novelty.

  • W2. Comparisons with other post-training methods for open-ended tasks remain limited. The paper includes CPT, SFT, and internal ablations, but comparisons with the closest self-rewarding and rubric-based RL approaches are mainly qualitative. Differences in supervision constrain direct comparisons, leaving the current results insufficient to fully establish POP's relative advantages over these approaches.

  • W3. Ablation evidence for the role of pretraining text is limited to healthcare. In Figure 5, POP and Eval w/o D have overall scores of approximately 30 and 29, respectively. Directly providing the source document during rubric generation yields a modest overall gain, with mixed results across scoring dimensions. Removing documents from the entire pipeline (w/o D) reduces the overall score to about 27, supporting their value to the full pipeline. However, these ablations cover only HealthBench500. Corresponding experiments on creative writing and instruction following are not reported, leaving it unclear whether this particular design provides similar benefits on those tasks.

Questions for authors

See Weaknesses.

2 comments

[W2] Comparisons with other post-training methods for open-ended tasks remain limited.

Although a fair comparison is difficult, we compare POP against the closest counterpart, SAO [1]. It synthesizes questions and responses and ranks the responses, all with the same LLM. The question-response pairs are used to train the LLM via preference optimization, similar to ours. The major difference between SAO and POP is that SAO uses strong LLM supervision for corpus curation. SAO creates questions from Persona Hub [2]. Persona Hub is a heavily compressed and reformatted version of raw web text, where the compression and reformatting are done by strong LLMs such as GPT-4.

We adapted SAO’s data synthesis and training pipeline to the three task settings that we use. We also controlled the final training set size to be similar to ours (SAO synthesizes 2935/1119/2935 preference pairs for healthcare/creative writing/instruction following, respectively). We show the results on Qwen-2.5-7B-Base as follows.

HealthBench500CreativeWritingV3IFEvalArenaHard
Qwen-2.5-7B26.71±0.3326.71\pm0.3342.31±0.4742.31\pm0.4731.79±0.5231.79\pm0.5212.26±0.1912.26\pm0.19
w/ SAO30.28±0.18\mathbf{30.28\pm0.18}39.41±0.5339.41\pm0.5331.12±0.7431.12\pm0.747.82±0.127.82\pm0.12
w/ POP29.64±0.4329.64\pm0.4344.31±0.71\mathbf{44.31\pm0.71}36.72±0.70\mathbf{36.72\pm0.70}16.20±0.17\mathbf{16.20\pm0.17}

Despite using fewer supervisions (SAO uses stronger LLM supervisions for corpus curation, but POP does not), POP outperforms SAO on creative writing and instruction following tasks. On healthcare QA, POP slightly underperforms SAO by 0.74%.

[1] Yin et al. “Aligning Large Language Models via Fully Self-Synthetic Data”. ACL, 2026. [2] Ge et al. “Scaling Synthetic Data Creation with 1,000,000,000 Personas”. arXiv preprint arXiv:2406.20094.

Dear Reviewer,

With the author response window closing soon, we would deeply appreciate it if you could go through our responses and reconsider your rating if they address your concerns.

Thank you again for your time and effort in reviewing our paper.

Best regards,

The Paper Authors.

Reviewer_W5TkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes POP, a self-play framework for post-training LLMs on open-ended tasks without relying on external supervision from humans or stronger models. POP uses the same model to generate queries and responses, construct evaluation rubrics grounded in pretraining text, and evaluate candidate responses. The highest and lowest scored responses are then selected to form preference pairs for DPO training. By grounding rubric generation in pretraining text, the method aims to improve evaluation reliability and mitigate reward hacking.

Critical strengths
  • The paper addresses a meaningful problem by extending autonomous self-play to open-ended tasks without relying on human annotations or stronger teacher models. The proposed framework is straightforward and easy to implement.

  • Experiments across healthcare QA, creative writing, and instruction following demonstrate consistent performance improvements. The ablation studies and response ranking analyses provide additional evidence for the effectiveness of the proposed approach.

Critical weaknesses
  • The proposed framework mainly combines existing techniques, including synthetic query generation, self-evaluation, preference pair construction, and DPO. Closely related works, such as Self-Rewarding Language Models [1], IPO [2], SAO [3], and Meta-Rewarding Language Models [4], have already explored similar self-improvement paradigms. The paper does not sufficiently clarify its novel contributions beyond introducing document-grounded rubrics.

  • The main experiments only compare POP with the original model, continued pretraining, and SFT, without evaluating against existing self-improvement methods. Representative methods such as Self-Rewarding Language Models [1], IPO [2], and SAO [3] could provide more meaningful comparisons under comparable data and computational budgets. Without such comparisons, it is difficult to assess the advantages of POP over prior self-improvement approaches.

[1] Self-Rewarding Language Models. ICML 2024.

[2] IPO: Your Language Model is Secretly a Preference Classifier. ACL 2025.

[3] Aligning Large Language Models via Fully Self-Synthetic Data. ACL 2026.

[4] Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. EMNLP 2025.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

1 comment

[W2] The main experiments only compare POP with the original model, continued pretraining, and SFT, without evaluating against existing self-improvement methods. Representative methods could provide more meaningful comparisons under comparable data and computational budgets. Without such comparisons, it is difficult to assess the advantages of POP over prior self-improvement approaches.

Although a fair comparison is difficult, we compare POP against the closest counterpart, SAO. We adapted SAO’s data synthesis and training pipeline to the three task settings that we use. We also controlled the final training set size to be similar to ours (SAO synthesizes 2935/1119/2935 preference pairs for healthcare/creative writing/instruction following, respectively). We show the results on Qwen-2.5-7B-Base as follows.

HealthBench500CreativeWritingV3IFEvalArenaHard
Qwen-2.5-7B26.71±0.3326.71\pm0.3342.31±0.4742.31\pm0.4731.79±0.5231.79\pm0.5212.26±0.1912.26\pm0.19
w/ SAO30.28±0.18\mathbf{30.28\pm0.18}39.41±0.5339.41\pm0.5331.12±0.7431.12\pm0.747.82±0.127.82\pm0.12
w/ POP29.64±0.4329.64\pm0.4344.31±0.71\mathbf{44.31\pm0.71}36.72±0.70\mathbf{36.72\pm0.70}16.20±0.17\mathbf{16.20\pm0.17}

Despite using fewer supervisions (SAO uses stronger LLM supervisions for corpus curation, but POP does not), POP outperforms SAO on creative writing and instruction-following tasks. On healthcare QA, POP slightly underperforms SAO by 0.74%.

Reviewer_4GNtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces POP that is an offline synthetic data generation and preference optimization pipeline designed to post-train LLMs on open-ended tasks without human annotations or stronger teacher models. POP first samples a passage (50-1024 words) from a domain-relevant pre-training corpus and prompts the model to synthesize a standalone query and a reference response. Second, it generates 32 candidate responses to the query without accessing the source passage. Third, conditioned on the source passage, query, reference answer, and 10 length-stratified candidate responses, the model generates a query-specific evaluation rubric and grades each candidate on a 0-to-2 scale. Finally, the highest and lowest scoring responses are paired to train the model for one epoch via DPO. The experiments show the effectiveness of this method.

Critical strengths
  1. Post-training is a surely important task, and training data is especially important under the scope of post-training.

  2. The paper has an insightful analysis of extreme pair selection under noise self-verification. It shows a transparent empirical breakdown of why self-rubrics can generate training signals under bad calibration, by evaluating reference model's rubric rankings against a strong teacher model and human curated rubrics.

  3. The pipeline is quite low-cost. It just uses a unified rubric generation and grading prompt across diverse domain by just requesting 12hrs on a A100.

Critical weaknesses
  1. Because the query generator sees the source text but the solver does not, the model frequently generates questions that reference missing passages (e.g., "Given the context of the paper..." in Fig. 29 and "Given the following context..." in Fig. 45). Without the source text, the solver is forced to guess, resulting in winning responses that hallucinate facts (such as naming the wrong enzymes in Fig. 32).

  2. The self-grader can be unreliable, as it awards a perfect length score to a ~450-word story that required 1,000 words (Fig. 40–41) and mistakenly treats a candidate's hallucination as part of the gold rubric (Fig. 50–51).

  3. While the paper claims that grounding rubrics in pre-training text is crucial to prevent reward hacking, the HealthBench500 ablations show very little benefit.

Questions for authors

In both the Healthcare QA and Instruction Following qualitative examples, the proposer violates the system prompt by generating questions that reference omitted text ("Given the context of the paper..." and "Given the following context..."), forcing the solver to guess missing facts. What percentage of synthesized queries suffer from this context leakage, and does training on winning responses that hallucinate facts (as in Figure 32) increase factual hallucination rates when evaluated outside of rubric-based formatting?

I'd also love to hear author's response to weaknesses session as well.

1 comment

Dear Reviewer,

With the author response window closing soon, we would deeply appreciate it if you could go through our responses and reconsider your rating if they address your concerns.

Thank you again for your time and effort in reviewing our paper.

Best regards,

The Paper Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.