acceptodds
Under review as a conference paper at ICLR 2027

Which Post Will Be More Engaging? Discovering Evaluation Criteria via LLM-based Reflection

Abstract

Predicting the relative popularity of social media posts plays an important role in understanding the relationship between content and user interactions, while informing content creation, candidate selection, and evaluative feedback for generative models. LLM-based evaluation and feedback-driven prompt optimization support text assessment and iterative instruction refinement, yet historical popularity outcomes provide only indirect guidance for revising evaluation criteria. The combined influence of content, creators, and platform distribution makes individual prediction errors an ambiguous basis for revision, increasing the risk of learning example-specific associations rather than reusable criteria. We introduce Grounded Reflection for Automated Criteria Exploration (GRACE), a framework that leverages LLM-based reflection to discover evaluation criteria from historical engagement feedback. GRACE examines discrepancies between model judgments and observed relative popularity across examples to refine and select topic-specific criteria for predicting the relative popularity of new posts. Across six topic-specific datasets, GRACE achieves 60.43% selective directional accuracy at 84.49% directional coverage, improving on the initial task prompt by 8.46 percentage points and outperforming the general prompt optimization methods evaluated in this study. These results support using historical engagement feedback to improve text-based relative popularity prediction, providing a methodological foundation for evaluation tools designed to aid content creation and selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
relative popularity prediction, social media, large language models, prompt optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ghKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses text to predict which social-media posts will be more popular. GRACE learns from past engagement errors to revise topic-specific rubrics, keeping changes that improve validation accuracy. At inference, it combines ten order-balanced judgments and abstains without enough agreement. On six Xiaohongshu topics, it improves selective directional accuracy and coverage. The appendix also reports a full-directional-accuracy gain, target-sensitivity analyses, another evaluator comparison, and exploratory Weibo cases.

Critical strengths
  1. The paper shows a clear, checkable way to link user feedback to rubric changes. It keeps reflection, editing, and validation separate, and Figure 4 shows how a revision changes how text cues are read. That helps us see what a popularity evaluator actually learns.

  2. The results clearly separate selective accuracy from coverage, full-set directional accuracy, exact-text splits, bootstrap uncertainty, and target sensitivity. The full-directional gain shows the improvement isn't just from abstaining. The stronger TF-IDF baseline and the unchanged-prompt Weibo case also help keep the claims in check.

Critical weaknesses
  1. I'm not sure what GRACE adds beyond standard prompt optimization. Error analysis, reflection, prompt editing, validation-based selection, and repeated judgments are familiar ingredients, and the experiments do not clearly show what the GRACE-specific design contributes. The method also requires repeated LLM calls while trailing TF-IDF on full-directional accuracy (55.71% vs. 62.91%). Readable rubrics could still be valuable, but the paper provides little evidence of the additional insight or practical benefit they offer.

  2. The engagement labels reflect both the post's content and the account's existing audience and exposure. Comparing posts within the same topic does not control for follower counts or creator popularity. Although the paper acknowledges these factors, it remains unclear how much they affect the learned rules. A post may receive more engagement because it comes from a popular account. Without a within-creator or follower-matched comparison, I find it difficult to interpret the rubrics as criteria for what makes the content itself engaging.

Questions for authors
  1. Could you provide an experiment showing the value of the learned rubrics beyond popularity prediction? For example, do they help people write or revise more engaging posts compared with generic writing guidance? An evaluation independent of GRACE itself would help establish their practical value.

  2. Could you report, within each topic, the Spearman correlation between the author's follower count and the post's normalized popularity score? This would help assess how strongly the target is associated with account popularity. Please also clarify when the follower counts were recorded relative to the engagement data.

1 comment

Thank you for the additional analyses. The low within-topic correlations and the improvement on follower-matched pairs substantially reduce my concern about account size. These results do not fully separate text from exposure, but I no longer see follower count as a main concern.

  1. The GRACE-specific methodological contribution remains unclear to me. Finding patterns across errors and selecting revisions on a validation set are familiar prompt-optimization practices. The comparisons support the overall workflow, but do not establish how much these particular design choices contribute.

  2. The revision example shows that the rubric can guide an edit, but not that it makes the post better. Organizing advice into a list is also common writing guidance, so the additional practical value of the learned criteria remains unclear. Given TF-IDF’s advantage in cost and full-directional accuracy, this still matters to my assessment of the contribution.

I appreciate the careful responses and have no further requests for experiments. However, these remaining concerns lead me to keep my original rating.

Reviewer_QG83OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GRACE, a framework designed to discover and refine evaluation criteria for predicting the relative popularity of social media posts by leveraging the reflective capabilities of LLMs on historical engagement data. This is a valuable and challenging problem, as post engagement is influenced by a complex interplay of content, creator influence, and platform distribution dynamics.

Main Contributions The paper proposes the GRACE framework, which iteratively refines an evaluation rubric by analyzing discrepancies between model predictions and observed user engagement. The framework establishes a closed loop involving an "evaluator-reflector-editor" to automate and ground the discovery of evaluation criteria.

Experiments conducted on six distinct topic-specific datasets demonstrate that GRACE significantly improves selective directional accuracy compared to an initial task prompt and other general-purpose prompt optimization methods.

Beyond predictive performance, the framework produces explicit and editable evaluation criteria, offering inspectable insights for content creators and platform analysts.

Critical strengths

Clear Problem Formulation and Innovative Method: The paper creatively frames the optimization of evaluation criteria as a data-driven search problem and proposes the GRACE framework as an effective solution. Instead of relying on manual trial-and-error to refine prompts, GRACE's "reflection" mechanism enables the LLM to learn from its mistakes and distill more reliable evaluation criteria, which is a novel approach.

Rigorous Experimental Design and Comprehensive Metrics: The experimental setup is robust. It not only compares performance across different methods but also considers a selective prediction scenario by using selective accuracy and coverage as key metrics, which is highly relevant for practical applications. Furthermore, the paper includes detailed sensitivity and ablation studies, such as analyzing the impact of different engagement signal weights and various evaluator models, which strengthens the credibility of its conclusions.

Insightful Research Direction: This work serves as an excellent example of how to leverage LLMs to learn and extract knowledge from noisy historical data. The core concept of using "reflection" and "iteration" to refine evaluation criteria is not limited to social media popularity prediction and could potentially be extended to other domains requiring subjective assessment, such as content recommendation and text quality evaluation.

Critical weaknesses

Novelty Discussion Could Be Strengthened: While the GRACE framework as a whole is new, its core components, such as prompt optimization and using LLMs as evaluators, have been explored in recent literature. The paper cites related works like ProTeGi and GEPA, but it needs to more clearly articulate the key differentiators and advantages of GRACE. Specifically, the distinction between "discovering" entirely new criteria from historical data versus simply "optimizing" an existing prompt could be emphasized more strongly.

Limited Discussion on Handling Data Noise: User engagement data from social media is inherently noisy, affected by factors like spam accounts, bots, and the platform's own distribution algorithm. While the paper acknowledges these factors, the methodology does not seem to explicitly model or control for this noise. It relies on the LLM's reflective ability to implicitly filter it. A key concern is whether GRACE remains effective when the noise level is high and whether mechanisms exist to assess or mitigate the impact of noise on the criteria discovery process.

Generalizability Needs Further Validation: The experiments are conducted on six datasets from a single social media platform (Xiaohongshu). This raises questions about how well the findings and the learned rubrics might generalize to other platforms like Weibo or Twitter, which have different community cultures, user behaviors, and content formats.

Cost and Efficiency Concerns: The GRACE framework involves numerous LLM calls for evaluation, reflection, and editing, particularly at inference, where repeated judgments are used to ensure reliability. This approach can be computationally expensive. The paper lacks a discussion on the cost-effectiveness and efficiency of the proposed method.

Questions for authors

In the "reflector" module, how does the LLM distinguish between incidental prediction errors and systematic flaws in the evaluation rubric? Is there a risk of the reflection process converging to a local optimum or even diverging?

The results show that TF-IDF logistic regression outperforms GRACE on full-set directional accuracy. What does this suggest about the scenarios where traditional, lexical-based models might still have an advantage over LLM-based methods? How do you interpret this finding?

To what extent are the evaluation criteria discovered by GRACE transferable? For example, could a rubric learned from a topic like "workplace bullying" be effectively used as an initial prompt for a related topic, such as "family relationships"?

In a real-world application, posts often contain multimodal content like images and videos. How would the GRACE framework be adapted to handle such content? How would the evaluation criteria for text be integrated with those for multimodal elements?

Reviewer_Guj6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how historical engagement data can be leveraged to improve the prediction of the relative popularity of social media posts and proposes a framework called Grounded Reflection for Automated Criteria Exploration (GRACE). The method transforms the numbers of likes, comments, and saves received by two posts on the same topic into relative-popularity labels. Discrepancies between model predictions and observed engagement outcomes are then used as feedback: an LLM-based reflector identifies recurring errors across examples and proposes topic-specific revisions to the evaluation criteria, while a prompt editor incorporates these revisions into the rubric. An independent validation set is subsequently used to determine whether each revision should be accepted. The proposed method achieves strong performance across multiple experiments.

Critical strengths
  1. GRACE uses discrepancies between model predictions and observed engagement outcomes as feedback. An LLM-based reflector identifies recurring issues across multiple erroneous examples and subsequently makes targeted revisions to the topic-specific evaluation criteria. This design not only reduces the risk of overfitting to individual examples but also preserves a complete record of criterion revisions, allowing researchers to examine how historical engagement feedback is translated into concrete and editable evaluation rules.

  2. The paper provides a reasonably comprehensive empirical evaluation. It compares the proposed method with several supervised-learning baselines on datasets covering six different topics and further investigates the influence of using different LLM evaluators. The results provide empirical evidence supporting the effectiveness of the proposed method.

Critical weaknesses
  1. The paper lacks comparisons with several important recent baselines, such as AutoCalibrate, AgentCF++, and OPRO. Consequently, it remains unclear whether the proposed method is genuinely more effective than existing approaches.

  2. Even among the methods already included in the comparison, the TF-IDF baseline appears to achieve higher accuracy while incurring substantially lower training and inference costs. Moreover, the experiments do not control for differences in the feedback provided to each method, model capability, or computational budget. It is therefore difficult to attribute the reported improvements specifically to GRACE’s grounded-reflection mechanism.

  3. The engagement labels do not control for differences in observation duration, creator influence, or platform exposure. Consequently, the resulting evaluation criteria may merely constitute post hoc rationalizations of confounded engagement outcomes rather than reliable textual factors that predict or drive popularity.

Questions for authors

See the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.