- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
OranSim is a pre-launch simulator for social-media marketing campaigns on RedNote. A campaign (creative, creator, targeting, budget) selects consumers from a synthetic population of 100,000 using a multiplicative exposure score. A hand-specified logistic model gives each selected consumer's click and engagement probability, and a 60-segment Hawkes process propagates the response. Separately, LightGBM regressors trained on 39,000 private RedNote notes predict reads, likes, collects and comments, and a lexicographic rule combines the two to choose a campaign. The evaluation covers the LightGBM predictors, one synthetic skincare campaign, off-policy and uplift estimators on public datasets, and a synthetic counterfactual test.
Critical strengths
- The framework is interpretable. It links each marketing action to consumer exposure, individual response and population propagation. Because candidate campaigns share the population and the random numbers, they can be compared directly and the comparison can be reproduced.
Critical weaknesses
-
The simulator is never validated against real consumer or campaign data, and its main result follows from its design. The population, exposure factors, logistic coefficients and Hawkes parameters are all set by hand (Table A.1), and the authors acknowledge that calibrating them would need campaign-level observations they do not have (Sec. 6). With these choices the headline finding (doubling the budget gives 1.96 times the response) is close to arithmetic: reach doubles by Eq. 1, and App. A.4 shows that total organic response is a fixed multiple of the paid input, whatever the influence matrix A. Replacing individual features by the audience mean changes M14 by less than 0.2% (Table 3), so consumer heterogeneity (Big Five, response state) adds little beyond who gets selected. Content vectors are hashed titles with no semantic relation to interests, so creatives A and B differ in M14 by under 0.1%, and the choice of creative is decided by LightGBM alone.
-
The experiments do not test OranSim as a system, and it is unclear what the population rollout contributes to the decision. Sections 5.3 and 5.4 evaluate standard off-policy estimators and uplift learners on KuaiRand, Open Bandit and X5, none of which feed the rollout, plus a synthetic generator the authors built themselves. There is no baseline for the full framework, such as LightGBM or a market-response model on its own, or an existing social simulator. Under the collect-first objective the final choice rests on LightGBM, and the rollout only orders options that differ in budget alone. The predictors also transfer poorly over time (latest-time of 0.25 to 0.36, and −0.01 for comments) and under-predict counts by 57% to 82% in the structured holdouts.
Questions for authors
- Can you compare the rollout's predicted response with real campaign outcomes (for example paid RedNote campaigns with known budgets), and report the error?
- What does the rollout add to the decision beyond LightGBM? An ablation that selects campaigns with LightGBM alone, or with a simple reach × mean-propensity model, would show this. Does LightGBM beat simpler predictors?
- Predicted engagement does not depend on budget by construction ( excludes it), so two campaigns with the same content get the same predicted collects at any budget. How should an advertiser use these predictions when deciding on a budget?
- Where do the demographic marginals (A.1) and the RedNote audience priors (A.2) come from, and how does the synthetic population compare with RedNote's actual user distribution?
- The hashed content vectors have no semantic relation to the interest vectors (Sec. 6). What, then, does content matching measure in the rollout?
- What is the basis for the logistic coefficients, the 0.5 engagement weight, the targeting strength of 2.0, the platform priors, and , ? App. B.7 varies them one at a time; were any of them fitted to data?
- Please add fold-level variance for Table 1.
5 comments
Response to Reviewer RXJh (2/3)
W1: the 1.96 ratio. Your derivation is right, and it holds exactly. Total 14-day response is M₁₄ = w·K·m̄·F: K is reach, m̄ the mean initial response of the consumers that exposure ranking selects, w = 20 the population weight, and F = 1.7575 the propagation multiplier, which depends only on the paid schedule, β, r and the time step because A is row-normalized. The identity reproduces all 450 rollouts (3 populations × 30 seeds × 5 options) with relative error below 6×10⁻⁸. So 1.96 = 2.00 × 0.981: reach doubles, and the 41,667 added consumers, ranked lower by exposure score, respond less on average. §5.2 traces this composition effect (contribution 2: doubling the budget "expands reach, changes audience composition"). The revision writes the identity in App. A.4 and gives the 0.981 factor in §5.2.
W1: heterogeneity. §5.2 reaches the same reading: "Heterogeneity thus shapes aggregate response mainly through which consumers the exposure ranking selects." Table 3 isolates this channel, +4.4% to +6.6% over uniform selection and under 0.2% from substituting mean features, and the closed form above makes it exact for total response.
Q2 and W2: what the rollout adds to the decision. The rollout handles the two actions for which the predictor has no input, budget and targeting: LightGBM gives s_b the same prediction as s₀, and s_cb the same as s_c. We ran the two ablations you propose, on 3 populations × 30 seeds; the comparison of LightGBM with simpler predictors is in (3/3).
| Model | Error in total M₁₄ | Ranking by M₁₄ | Ranking by M₁₄ per CNY |
|---|---|---|---|
| Reach scaling from s₀ | 0 to +1.93% | options with the same budget tie | all options tie |
| Reach × population-average response | −4.2% to −6.2% | same top option, order differs | different top option; budget pairs tie |
| Exposure ranking + initial responses, closed-form F | < 6×10⁻⁸ | identical | identical |
The part of the rollout that changes decisions is the exposure ranking together with the individual responses of the consumers it selects. Removing it, as the population-average model does, makes response linear in reach, so under the response-per-CNY objective every budget pair ties and the model cannot choose a budget. Propagation sets the multiplier F and distributes response across segments and days: the cumulative segment shares of response differ from the selected audience's composition by a total-variation distance of 0.05–0.10, and replacing A with uniform weights moves the targeted segments' organic share from 96.1% to 94.3%.
Under the collect-first objective, ordering by predicted collects and then by budget gives the same ranking as Eq. 7 (s_k > s_b > s₀ > s_cb > s_c). As §5.2 states, the predictor chooses creative and creator there, and the rollout orders the budget levels. Under response objectives the rollout chooses: s_cb by M₁₄ and s₀ by M₁₄ per CNY.
On the paid campaign, the two models you name are the baselines in the general response (item 1). For the held-out week, LightGBM alone predicts 1.26 times the observed engaged views and reach × mean propensity 2.05 times, against 1.08 for the rollout, which takes each ad group's targeting and spend. The rollout's creative-level error is below LightGBM's by 0.08 of the observed total (95% interval 0.02–0.15).
Q3: budget and the predictor. ψ excludes b because the historical notes do not record promotion spend: the predictor estimates engagement from content, creator and timing, and the budget enters through reach in the rollout. An advertiser uses the predictor to choose content and creator and the rollout to choose the budget level. In the case, doubling the budget raises M₁₄ by 96.2% and lowers response per CNY by 1.9%; the size of this curvature is set by the response slopes, which are design values (Q6). The revision writes this division into §3.5, next to Eq. 7.
Response to Reviewer RXJh (3/3)
Q5 and W1: content matching. As §6 states, the rollout's content vectors are hashed titles, so content match in the rollout has no semantic relation to interests, and creatives A and B differ by less than 0.1% in M₁₄. In the case, creative choice is made by the predictor, which embeds title and body with text-embedding-3-small and whose temporal and grouped results are under Q7 below. The content-matching term in Eq. 1 is where an interest model with semantic vectors enters exposure ranking. The revision states in §3.2 and §5.2 that creative differences in the case come from the predictor, and that the rollout's content channel uses hashed vectors.
Q4: population sources. Age band, gender and city-tier marginals approximate China's Seventh National Census (2020). Platform penetration and audience priors approximate QuestMobile's 2024 reports, for example 65% female among RedNote users against the census share of 48.7%. App. A.1–A.2 will list every marginal and prior with its source. Against the follower profiles of 12,983 RedNote creators, the simulated RedNote users are 81.0–81.3% female against 80.8–83.4% (total-variation distance 0.005); QuestMobile's 65% counts all users, while the profiles count users who follow creators. The simulated age distribution is older: under 25 / 25–34 / 35–44 / 45+ is 21.0 / 26.7 / 30.0 / 22.4% against 37.4 / 33.8 / 19.4 / 9.5% (distance 0.235). The revision reports the age deviation in §6 and gives the profile marginals in App. A.1.
W2: experiments and framework baselines. §5.3 and §5.4 evaluate the estimators a campaign comparison uses when logs exist: off-policy estimators for content allocation, treatment-effect learners for audience selection, and predictor-based counterfactual scoring. They do not run the rollout. The revision adds the experiment–component table from the general response to §4. Framework-level baselines are the predictor alone and reach × population-average response, both under Q2. The paid campaign has spend and outcome series, so a market-response model joins these two baselines there (general response, item 1). As §1 notes, the social simulators we cite do not take creative, creator, targeting and budget as actions, so a comparison with them would first require adding this action space; the two decompositions in Q2 isolate what the rollout contributes.
Q7, the LightGBM part of Q2, and transfer over time (W2). Fold standard deviations: with Table 1's row split, 0.007–0.012; with folds grouped by note, R² is 0.504 / 0.539 / 0.482 / 0.419 (reads / likes / collects / comments) with standard deviations of 0.013–0.018. The revision uses the grouped protocol in Table 1 and the abstract. Simpler predictors on the same grouped folds:
| Predictor | Reads | Likes | Collects | Comments |
|---|---|---|---|---|
| Training mean | −0.000 | −0.000 | −0.000 | −0.000 |
| log follower count, linear | 0.005 | 0.061 | 0.054 | 0.021 |
| Metadata only, LightGBM | 0.429 | 0.452 | 0.389 | 0.276 |
| Text embeddings only, LightGBM | 0.260 | 0.281 | 0.266 | 0.284 |
| Full predictor | 0.504 | 0.539 | 0.482 | 0.419 |
On the note-disjoint temporal split of the 39k corpus, the full predictor reaches R² 0.562 / 0.575 / 0.503 / 0.443 and Spearman ρ 0.671–0.714, and it is higher than every baseline above and the creator's historical median on all four counts. The temporal row of Table 1 uses the separate 12k corpus with hashed text; there, comment R² of −0.007 is measured against the test-period mean, which is unavailable at deployment, and the training-period baselines score −1.01 to −5.23 on the same split. Back-transformed counts remain below observed means (44–59% of observed on the 39k temporal split); the collect-first rule uses only their order, as §6 states.
Response to Reviewer RXJh (1/3)
Thank you for working through Eq. 1 and App. A.4 in this detail. W1 and W2 below refer to the two paragraphs of your weaknesses.
Q1 and W1: comparison with campaign outcomes. We ran this comparison on a paid TikTok campaign with known daily spend per creative (general response, item 1). With response intercepts and spend-to-reach conversion re-estimated on two weeks, the rollout predicts 1.08 times the observed engaged views of the held-out week (95% interval 0.98–1.21), with a creative-level absolute error of 0.26 of the observed total; its ratios for likes and 2-second views are 0.92 and 0.94. Reach × mean propensity gives 2.05 and 1.05, and LightGBM alone 1.26 and 0.34. The slopes, β and r in this test are the paper's design values. The components were also tested one at a time:
The revision adds the campaign test as §5.5 and this table to App. A, and rewrites §6 to state which parts are re-estimated on the campaign, which are compared with observed data, and which remain design values.
Q6. Basis of the parameters. The logistic coefficients, the 0.5 engagement weight, the 2.0 targeting strength, the platform priors, β and r are design values; App. B.7 varies them one at a time, and the paper fits none of them. The new experiments fit the two parts that public or internal data can identify.
The intercepts, which set absolute response levels, are identified from randomized exposure, and the same logistic structure takes the viewing history and video attributes without change. The revision reports this fit in App. A.3 and labels the slopes in Table A.1 as design values.