acceptodds
Under review as a conference paper at ICLR 2027

The Crowd Already Votes: Learning Creative-Writing Rewards from Human Interaction

Abstract

Creative writing aims to produce automatically open-ended text that is coherent and engaging narratives. Despite their strong text generation capabilities, large language models (LLMs) still struggle to consistently produce writing that receives high evaluations from human audiences. A central obstacle is that writing quality is highly subjective and real-world judgments emerge from collective rather than individual preferences. Consequently, existing models poorly approximate reader responses. We introduce REALFEEDBACK, a data-construction paradigm that treats online communities as continuously operating sources of preference supervision. It converts naturally occurring human interactions into reward signals without new large-scale annotation. We instantiate REALFEEDBACK in two domains: (1) For story generation, inline critiques yield passage-level process labels. (2) For forum discussions, nested discussions yield outcome-level preference pairs. We use these data to train REALFEEDBACK-PRM and REALFEEDBACK-ORM. Across established benchmarks and our constructed test sets, REALFEEDBACK-REWARD achieves state-of-the-art performance and outperforms much larger models. We also adopt REALFEEDBACK-REWARD in downstream test-time scaling applications for best-of-n (BoN) selection and find that it always selects outputs that receive higher human preference scores. These results position online communities as sustainable infrastructure for training reward models that better reflect collective human judgment.

Primary area:
datasets and benchmarks
Keywords:
Creative Writing, Reinforcement Learning, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2tFUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Authors introduce REALFEEDBACK — a data-construction paradigm that treats online communities as continuously operating sources of preference supervision. It converts naturally occurring human interactions into reward signals without new largescale annotation. The central research question is how to systematically transform heterogeneous social interactions into reward supervision. The authors collect user feedback data from two communities: Critique Circle (evaluation of individual story fragments) and Baidu Tieba (evaluation of engagement in forum discussions). A Process Reward Model (PRM) is trained on 368 thousand fragment annotations, and an Outcome Reward Model (ORM) is trained on 143 thousand preference pairs. For evaluation, REALFEEDBACK-BENCH was created with 1,500 stories and 1,500 forum discussions; the models outperform existing reward models on most metrics and show the best results in Best-of-N story selection. The key takeaway: it is important not simply to use behavioral data, but to correctly interpret user interactions, since the number of replies alone does not allow reliably predicting preferences.

Critical strengths

The idea is intuitive, paper is well-structured and clear, the data collection methodology and the training phases are described in details.

Critical weaknesses

In the proposed approach to building the PRM, the reward signal is effectively reduced to the sentiment of comments with a minor heuristic correction. However, the evaluation of creative texts is multidimensional and includes many criteria: originality, composition, character development, style, emotional impact, and other aspects. It is not obvious how well such a simplified signal actually reflects text quality and is capable of accounting for different aspects of creative work. Accordingly, the question remains open as to how transferable the proposed approach is to other criteria for evaluating creative content.

A very detailed pipeline is described, however the data themselves are closed.

Please add confidence intervals and paired significance tests.

Questions for authors

Was agreement among the human annotators themselves measured?

Formulas 3 and 4 are confusing in the aggregation strategy. Could the authors clarify which formulation was used for LitBench, StoryRMB, and the Best-of-N experiments?

Reviewer_db12OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is a paper on data construction for reward modeling. The paper quantifies the inline comments of CritiqueCircle into 368,230 paragraph-level process labels (VADER sentiment + correction of criticism keywords + majority voting across commenters), and quantifies the nested replies on Baidu Tieba as 143,402 result-level preference pairs based on "reply - floor cosine similarity". It trains 8B PRM and ORM separately, and builds its own RF-Bench for evaluation, supplemented by 500 pairs of manual audits (consistency rate 80.65%) and downstream validation under the Best-of-N approach.

Critical strengths

The quantification method ablation (Table 4) achieved an acceptable level of strength: the number of response labels was only 54.2/50.9 (close to random), proving that the benefits came from the design of the quantifier rather than "any behavior data is sufficient", which isolated the key factor. The cross-criteria consensus ablation (from 46.3 on StoryRMB to 71.6) also provided incremental evidence.

Critical weaknesses
  1. The boundaries between the core novelty and the existing work have not been clearly defined. LitBench has transformed Reddit likes into preference pairs. The substantive increment of this paper is "switching to a different platform, using a different heuristic quantifier, and adding a layer of confidence filtering", which belongs to data engineering rather than paradigm innovation.

  2. The statement "Online communities are the supervisory infrastructure" goes beyond the scope that can be supported by two examples. More seriously, there is a self-circular evaluation: the tags of the RF-Bench forum track are generated by the same quantifier as the training data, and the 71.9% of the model's measurement on its own benchmark is measuring the fit of the Q function rather than human preferences; the cross-platform migration of Reddit alleviates this problem to some extent, but its tags (net voting numbers) are from a different source than the training signal, and the two benchmarks do not distinguish whether they are measuring "human preferences" or "heuristics of each platform", and the paper does not make such a distinction.

  3. The key constants such as the VADER penalty coefficient 0.5, the Jaccard threshold 0.5, and the Δ stratification boundary 0.050/0.110/0.216 have no sensitivity analysis; the manual audit is only 500 pairs and limited to a single community, which is insufficient to support the title-level conclusion of "tracking collective human judgment".

  4. The experiments are not solid.

Questions for authors

1.How do REALFEEDBACK-ORM and REALFEEDBACK-PRM perform on public, off-the-shelf reward-model benchmarks such as RewardBench and RM-Bench? Do the gains from community-derived supervision hold outside benchmarks that are self-built or topically adjacent to the training communities?

2.If the same 8B model is trained on LitBench's public training split — which is likewise derived from community votes — with identical architecture, hyperparameters, and data budget, does REALFEEDBACK-PRM still win across all benchmarks? In other words, does the improvement come from the interaction quantifier itself, or merely from a closer match between training and evaluation distributions?

3.How does accuracy on the already-used public benchmarks (Reddit, LitBench, StoryRMB) change when the models are trained on 10%, 25%, 50%, and 100% of the supervision pools? Does the curve still rise at full scale, as the "sustainable infrastructure" claim requires, or does it saturate early?

4.Are the main results in Tables 2–3 statistically significant under bootstrap confidence intervals or multi-seed variance? Specifically, is the 0.8-point Reddit gap between REALFEEDBACK-ORM (74.6%) and GPT-5.5 (75.4%) — the paper's only headline loss — a real difference or within noise on 1,500 instances?

Reviewer_8ir5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This is an interesting paper in the way that authors use distance supervision from two kind of forums to build both process reward models and outcome reward models to judge writing. They create REALFEEDBACK. And then they train these models and evaluate them on a benchmark that they create from the same domain, but also on Reddit, and they compare it with several state-of-the-art reward models as well, and then even with Claude zero-shot. And then they show that their model is actually the most superior on that, including also some other benchmarks like LitBench, and they also do some human eval.

Critical strengths

Good intuitive idea to use distant supervision to build better reward models

Critical weaknesses

No expert eval No eval on WQ benchmark to which the work bears substantial similarity LITBench is noisy. Community signal is not what needs to be take seriously for alignment in creative writing. So the motivation is noisy and the results are so unreliable

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.