- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines forecast-informed services in a simulated data market with buyers and sellers using llms. It uses five prediction interfaces to cover different aspects like choices, action outcomes, task gains, and seller trajectories. The evaluation is divided into three parts: predictive accuracy, fixed-state decision studies, and ongoing market interventions. The most significant finding comes from a 72-blueprint online study, which shows that customizing a buyer-value package using quantile LightGBM boosts the buyer’s net technical surplus by 41.16 accounting units over 42 days compared to using only reference value information, with a 95% confidence interval of [27.68, 55.08]. Another study that provides descriptive advice does not show any clear benefits. Compared to not receiving any advice, the value service increases the buyer’s surplus but decreases the number of transactions and the seller’s gross revenue.
Critical strengths
-
The evaluation makes useful distinctions between predictor replacement under a fixed rule, information replacement at fixed states, and interventions in continuing markets.
-
Separating predictive quality, estimator substitution under a fixed rule, and online service effects and refusing to let one stand in for the other is a good evaluation design.
Critical weaknesses
-
External validity is thin, and the headline number doesn't map to anything real. The buyers are LLM agents with no validation that their bargaining or purchase behavior resembles any market participant; task gain is logistic-regression loss reduction on small classification datasets; money sits on an "experimental accounting scale" initialized from training gains.
-
The prospective information-service implementation remains insufficiently verified. Historical temporal evaluation conditions on actual trajectory-arrival capacities, while their mapping to online temporal inputs is incomplete.
-
The paper studies the Bayesian persuasion setting without engaging the literature that defined it
Questions for authors
-
For the executed E3 runs, what are the exact post-purchase data-update, repeat-purchase, and utility-cache rules? Can a transaction-level trace establish which buyer snapshot each gain used, and clarify whether these rules affect the interpretation or ranking of the reported accounting totals?
-
How sensitive are the E3 surplus contrasts to that initialization — could the VIND advantage shrink or reverse under a different anchoring?
Response to Reviewer StWG Part1
Thank you for recognizing the value of separating predictive quality, fixed-state decisions, and continuing-market effects. We have strengthened the revised manuscript with additional price-sensitivity experiments, cumulative-acquisition evaluations, execution audits, and a clearer discussion of economic interpretation and information design.
1. E3 execution and cumulative utility
We independently reproduced the original E3 buyer-surplus effects from saved market records: VIND − VREF = +41.16 (95% CI [27.68, 55.08]) and VIND − N0 = +46.44 ([29.35, 64.68]), both with Holm-adjusted p = 0.00002.
The expanded transaction-level audit confirms that recorded gains use fixed buyer–product lookup values, including repeated purchases. It also identifies the remaining limitations in historical buyer-state, delivered-data, and cache provenance. We now make the transaction-accounting interpretation explicit (Sections H.3–H.5).
To investigate actual acquisition outcomes, we additionally implemented explicit training-data and ownership updates, once-per-buyer product-version purchases, classifier refitting, and independent terminal evaluation (Appendix A). These new experiments provide complementary evidence on cumulative learning without conflating their endpoints with original E3 accounting.
2. Price initialization and sensitivity
We clarified the original price construction, which uses training-only positive task gains to initialize the experimental price scale (Section A.5).
We also conducted a new 48-root price intervention with three price worlds, each executing N0, VREF, and VIND under cumulative acquisition.
Table 1. Buyer-only price sensitivity: terminal net-surplus contrasts (Table G.3).
Individualized information improves terminal surplus over references within all three evaluated buyer-only price worlds. However, the price-interaction contrasts do not pass their primary Holm-2 criterion (both adjusted p = 0.576). The lowest-price world also reaches the quote floor.
Importantly, these experiments vary prices in buyer-only acquisition markets, not the original E3 bilateral launch-price mechanism. They provide additional sensitivity evidence but do not establish price invariance or rule out a smaller or reversed original E3 effect under alternative initializations.
3. Prospective information delivery
We strengthened prospective implementation evidence through a new 72-root bilateral seller-information experiment.
The service uses frozen Core14/S1 models, public decision-time inputs, and arrival conditions sampled from the known mechanism rather than realized future events. A future-event perturbation check verifies invariant predictor inputs and outputs under fixed public information and prediction seeds (Section B.8).
Relative to training references, individualized seller forecasts reduce seller gross revenue by 7.14 and increase buyer terminal net surplus by 7.36 (both primary Holm-adjusted p = 0.0002). Most of the buyer-surplus improvement reflects lower payments, not an established learning improvement (Table 4).
The historical E3 temporal input-to-checkpoint bindings remain incomplete, and Controlled recovery supports partial member-level reweighting rather than full historical recalibration. The prospective experiment provides separately verified implementation evidence, not a retrospective reconstruction of the original execution.
4. Practical relevance and related work
Our target setting is data-market platforms serving autonomous software agents, rather than LLMs acting as validated proxies for human traders. Task utility is measured independently through classifier performance, while prices remain experimental accounting quantities rather than real monetary valuations.
We expanded acquisition evaluation to Bank Marketing and MAGIC with logistic regression and extra trees (Appendix G). The heterogeneous results, including negative net surplus and strong numerical-rule baselines, clarify the limits of generalization.
We also added the foundational Bayesian persuasion and information-design literature (Kamenica & Gentzkow, 2011; Bergemann & Morris, 2019). DIM-WM evaluates predefined forecast-information packages and observed agent decisions, rather than optimizing sender signaling or assuming Bayesian-optimal receivers (Section 6).
Overall, the revision preserves the original E3 findings while providing stronger evidence on price sensitivity, cumulative acquisition, prospective information delivery, and the boundaries of practical interpretation.