acceptodds
Withdrawn from ICLR 2027

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training models for open-ended writing requires diverse prompts and sufficiently precise specifications of response quality. Constructing such specifications at scale entails recovering task requirements, consolidating overlapping criteria, and defining observable differences between performance levels. We introduce Writer-R1, a writing training pipeline built around Inductive Rubric Generation (IRG), which integrates these operations into an automated rubric construction procedure. The pipeline expands an open writing corpus and uses the resulting prompt-specific rubrics throughout supervised training and reinforcement learning (RL). Each rubric is concatenated with the user request and also defines the reward criteria, making the requirements used for assessment available to the writer before generation. A compact model, RubricGen-4B, learns to supply rubrics for new requests. Writer-R1 improves WritingBench scores by 0.182 and 0.158 over the 4B and 8B backbones, respectively, with gains also on HelloBench. Training ablations favor IRG over direct rubric generation, and further analyses support rubric transfer and the robustness of writer improvements across evaluators. These results support using inductively mined rubrics as both generation guidance and reward specifications for open-ended writing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Dynamic rubric mining, Open-ended writing, Reinforcement learning, Reward modeling, LLM evaluation, Long-form text generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SDVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Writer-R1 addresses the challenge of defining evaluation criteria and rewards for open-ended writing. Its Inductive Rubric Generation (IRG) procedure separates rubric construction into requirement discovery, dimension consolidation, and scoring specification. The resulting rubrics guide generation, select SFT demonstrations, and define RL rewards. At inference time, a compact rubric generator produces task-specific criteria for the writer. The reported improvements, particularly over direct rubric generation, suggest that the quality of the assessment criteria matters substantially for training.

My main concern is the pipeline’s dependence on the initial discovery of candidate evaluation dimensions. If this step misses important requirements or introduces unwarranted stylistic preferences, subsequent consolidation and scoring descriptions may not correct those errors.

Critical strengths
  • Well-motivated rubric design. Case-specific rubrics are a suitable choice for open-ended writing, where quality depends on the request, audience, genre, and explicit constraints. Consolidating overlapping fine-grained dimensions from candidate proposals into distinct evaluation dimensions is intuitively sound: it reduces redundant scoring while preserving task-specific requirements. Defining assessment scopes and performance levels also makes the criteria more concrete and actionable.

  • Empirical support from post-training. The SFT and RL experiments provide evidence that the resulting rubrics are useful both as generation guidance and as reward specifications. IRG consistently outperforms direct rubric generation in the training comparisons, and subsequent RL further improves performance. The gains over both the 4B and 8B backbones on WritingBench and HelloBench support the practical effectiveness of the overall pipeline.

Critical weaknesses
  • Insufficient analysis of the initial dimension proposals. Although the overall rubric construction pipeline is reasonable, it depends heavily on the quality of the model’s initial proposals. This important stage warrants more systematic experiments, varying the proposing model, prompt formulation, and number of candidate dimensions, and testing sensitivity to missing, redundant, or irrelevant criteria. Such choices can substantially affect the validity of the final rubrics and the resulting rewards. The current analysis does not sufficiently examine these variables or establish whether later consolidation can reliably correct errors introduced at this stage.

  • Unclear applicability to larger MoE models. While computational constraints may justify focusing the key ablations on 4B-scale models, it remains unclear whether the approach is effective for larger MoE models, such as those at the 35B scale. If post-training these models is too costly, directly evaluating their rollouts conditioned on framework-generated rubrics could offer a practical, lower-cost quantitative analysis. Although this would only assess the utility of inference-time rubric guidance, it would still provide useful evidence about the approach’s applicability beyond small models.

Questions for authors
  • Have you evaluated mixed-model configurations across requirement discovery, dimension consolidation, and scoring specification?

  • If so, which stages require an expensive frontier model, and which can use a more economical model without materially reducing rubric quality or downstream writing performance?

Reviewer_66peOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Writer-R1, a rubric-guided training framework for open-ended writing. It introduces Inductive Rubric Generation (IRG) to construct task-specific evaluation criteria, which are used for both generation guidance and RL rewards. Experiments demonstrate substantial improvements on WritingBench and HelloBench.

Critical strengths
  • Strong empirical results. The method achieves substantial improvements across two writing benchmarks, supported by ablation studies and evaluations from multiple LLM judges and human experts.

  • Practical framework. Using task-specific rubrics as both generation guidance and reward specifications provides a useful approach to improving open-ended writing.

Critical weaknesses
  • Limited methodological novelty. IRG primarily consists of multi-stage LLM prompting for rubric construction, combined with existing SFT and RL techniques. Its distinction from prior rubric-based training approaches is relatively incremental. WritingBench already generates rubrics for each prompt and the work applies the sample-wise rubrics for training.

  • Insufficient isolation of inference-time rubric conditioning. Writer-R1 receives additional rubrics generated by a separate model at inference time, while most baselines only receive the original prompts. It remains unclear how much improvement comes from better model training versus additional inference-time guidance.

  • Unmatched rubric construction costs. IRG involves multiple LLM calls, whereas the direct rubric generation baseline uses a single call. The reported improvement may partially result from additional computation rather than the proposed construction procedure itself.

Questions for authors
  • How much of the performance improvement remains without inference-time rubrics? What happens if the baseline models receive the same generated rubrics?

  • Can the authors compare IRG with direct rubric generation under matched inference and construction budgets?

Reviewer_jnpQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors achieve state-of-the-art performance on WritingBench and HelloBench through SFT followed by RL.

Critical strengths

The paper is well organized, and the training details are clearly documented in the appendices.

Critical weaknesses
  • As of late 2026, SFT followed by RL is already a well-established training recipe. The paper reads largely as a report of an engineering implementation, with very limited novelty.
  • The three steps of IRG appear to be hand-crafted heuristics, and no ablation studies establish the necessity of each component.
  • In Table 3, DRG + SFT performs worse than SFT alone, which undermines its usefulness as a baseline for evaluating IRG.
Questions for authors

The reported results are in-domain. What evidence shows that training does not cause catastrophic forgetting or reduced generalization, and that the model does not exhibit reward hacking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.