- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper uses text to predict which social-media posts will be more popular. GRACE learns from past engagement errors to revise topic-specific rubrics, keeping changes that improve validation accuracy. At inference, it combines ten order-balanced judgments and abstains without enough agreement. On six Xiaohongshu topics, it improves selective directional accuracy and coverage. The appendix also reports a full-directional-accuracy gain, target-sensitivity analyses, another evaluator comparison, and exploratory Weibo cases.
Critical strengths
-
The paper shows a clear, checkable way to link user feedback to rubric changes. It keeps reflection, editing, and validation separate, and Figure 4 shows how a revision changes how text cues are read. That helps us see what a popularity evaluator actually learns.
-
The results clearly separate selective accuracy from coverage, full-set directional accuracy, exact-text splits, bootstrap uncertainty, and target sensitivity. The full-directional gain shows the improvement isn't just from abstaining. The stronger TF-IDF baseline and the unchanged-prompt Weibo case also help keep the claims in check.
Critical weaknesses
-
I'm not sure what GRACE adds beyond standard prompt optimization. Error analysis, reflection, prompt editing, validation-based selection, and repeated judgments are familiar ingredients, and the experiments do not clearly show what the GRACE-specific design contributes. The method also requires repeated LLM calls while trailing TF-IDF on full-directional accuracy (55.71% vs. 62.91%). Readable rubrics could still be valuable, but the paper provides little evidence of the additional insight or practical benefit they offer.
-
The engagement labels reflect both the post's content and the account's existing audience and exposure. Comparing posts within the same topic does not control for follower counts or creator popularity. Although the paper acknowledges these factors, it remains unclear how much they affect the learned rules. A post may receive more engagement because it comes from a popular account. Without a within-creator or follower-matched comparison, I find it difficult to interpret the rubrics as criteria for what makes the content itself engaging.
Questions for authors
-
Could you provide an experiment showing the value of the learned rubrics beyond popularity prediction? For example, do they help people write or revise more engaging posts compared with generic writing guidance? An evaluation independent of GRACE itself would help establish their practical value.
-
Could you report, within each topic, the Spearman correlation between the author's follower count and the post's normalized popularity score? This would help assess how strongly the target is associated with account popularity. Please also clarify when the follower counts were recorded relative to the engagement data.
Thank you for the additional analyses. The low within-topic correlations and the improvement on follower-matched pairs substantially reduce my concern about account size. These results do not fully separate text from exposure, but I no longer see follower count as a main concern.
The GRACE-specific methodological contribution remains unclear to me. Finding patterns across errors and selecting revisions on a validation set are familiar prompt-optimization practices. The comparisons support the overall workflow, but do not establish how much these particular design choices contribute.
The revision example shows that the rubric can guide an edit, but not that it makes the post better. Organizing advice into a list is also common writing guidance, so the additional practical value of the learned criteria remains unclear. Given TF-IDF’s advantage in cost and full-directional accuracy, this still matters to my assessment of the contribution.
I appreciate the careful responses and have no further requests for experiments. However, these remaining concerns lead me to keep my original rating.