acceptodds
Under review as a conference paper at ICLR 2027

The World as a Curator: Learning from Market Outcomes for Financial Reasoning

Abstract

Supervised fine tuning is widely used to adapt large language models to domain tasks, traditionally relying on high quality examples annotated by experts. Limited coverage of existing datasets and costly expert annotation make expanding high quality domain supervision a persistent challenge. Synthetic instruction and response pairs offer a scalable alternative, but internal judgments cannot independently verify agreement with actual outcomes when generation and selection share model priors without external evidence, while external feedback is often delayed, noisy, and confounded. To this end, we propose World Informed Selection with Energy and Reliability (WISER), which selects responses for student fine tuning using historical market feedback and its reliability to improve future financial event reasoning. Specifically, we lock candidates before outcome access, train a World Feedback Energy model using nonoverlapping selection and confirmation windows to learn nonadditive interactions among candidate attributes, market compatibility, and feedback reliability, and select its configuration through constrained Bayesian optimization within 64 evaluations. Extensive results on 255 post freeze events across China, the United States, Europe, and Japan show gains of 0.024 in Entity NDCG@5, 0.027 in Direction MCC, and 0.10 in cost adjusted Sharpe over Direct WF Score, demonstrating that selected supervision improves future reasoning and investment performance using event time inputs alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Synthetic Instruction Data Curation, Large Language Model Fine-Tuning, Outcome-Guided Data Selection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ReRoOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using delayed market outcomes, instead of model-internal scores, to curate synthetic fine-tuning data for financial event reasoning. A frozen generator produces several candidate analyses per event, and a learned energy-based selector, trained to predict later market-confirmed preferences from early market signals and weighted by an event reliability score, picks one candidate for fine-tuning a student model. On a post-freeze test period across several markets, the authors report that the resulting student outperforms text-only data-selection methods, a fixed early-feedback rule, and financial baselines, and they add a theoretical appendix.

Critical strengths
  1. Leakage-aware, prospective protocol. Candidates are committed before outcomes are seen, windows are disjoint, the test is post-freeze, and pools and budgets are matched, so selection rules are compared cleanly.
  2. Outcome-based selection clearly helps. A simple early-market rule beats all text-only selectors, consistently across markets, periods and student models, though from a single test window.
Critical weaknesses
  1. The added complexity is not justified. A fixed early-feedback rule delivers most of the gain, learned variants without the full combination fall below it, and the selector's held-out loss barely beats a uniform predictor. The heavy notation and very long theory appendix, mostly standard identities, are not tied to any empirical claim and obscure the core idea. I hope it can be more concise and reuse symbols.
  2. Statistical support is missing. The promised paired bootstrap and multiple-testing corrections are absent and the reported spread seems to reflect seeds only, so small margins on one test window are not shown to be significant; this can be added from existing outputs.
Questions for authors
  1. Please report event-paired intervals and adjusted tests for the learned selector versus the fixed rule, and state what the reported spread measures.
  2. The fixed rule has lower ranking quality but much higher Sharpe than the learned variants; is this a portfolio-rule artifact rather than better reasoning?
  3. Can you provide timestamped manifests and a check for memorized post-event knowledge in the generator?
3 comments

We sincerely appreciate the reviewer’s thoughtful comments and constructive suggestions on methodological clarity, statistical evidence, portfolio evaluation, and auditability. Guided by this feedback, we clarify loss interpretation and theoretical connections, provide paired statistical analyses and portfolio sensitivity checks, and add generator memory probes and specifications for timestamped audit records to improve overall transparency.

W1 Selection Gains and Theoretical Clarity:

Thank you for asking about complexity and theoretical clarity. Under matched candidate pools and student budgets, WISER improves NDCG@5, MCC, and Sharpe by 0.024, 0.027, and 0.10 over Direct WF Score, and by 0.020, 0.023, and 0.58 over H0 under equal BO budgets of 64 evaluations, as reported in Tables 7 and 8. H0 with BO and default WFE also improve NDCG@5 and MCC over the fixed rule. Additional fitting and search occur offline. Equation 5 analytically marginalizes latent variables, giving scoring cost O(Kd(H+1))O(Kd(H+1)) given features and context biases. Student inference uses only event-time inputs.

Figure 4a reports a held-out CN diagnostic of 2.075, compared with uniform cross-entropy log⁡8≈2.079\log 8\approx2.079. Proposition E.11 decomposes per-event cross-entropy as

ℓe=H(Qe)+KL⁡!(Qe∥pθ(⋅∣e)).\ell_e=H(Q_e)+\operatorname{KL}\\!\left(Q_e\Vert p_{\theta}(\cdot\mid e)\right).

Target entropy determines the maximum improvement over uniform cross-entropy. Equation 6 adds regularization. We will report cross-entropy, target entropy, and regularization separately under identical events and weights. Figure 4a illustrates optimization; Table 7 evaluates selection quality and student outcomes.

Table 12 links Theorem E.6's reliability interactions to Table 7, Corollary E.21's confirmation-utility regret bound on fixed events and weights to Appendix C.1, and Theorem E.17's margin stability to Figure 4b. We will streamline Section 3.4 and Appendix E around these connections, consolidate standard derivations and definitions, retain assumptions, and consistently use ρe\rho_e, QeQ_e, and pθ(⋅∣e)p_\theta(\cdot\mid e) for event reliability, confirmation preferences, and selection probabilities. Questions 1 and 2 address paired statistics and portfolio construction.

W2 Statistical Support for Incremental Gains:

Thank you for requesting explicit statistical support. The “±” values denote sample standard deviations across five paired seeds. With trained runs fixed, bootstrap resamples share date blocks across methods and seeds to estimate test-period uncertainty in paired differences. Holm correction covers six tests across three primary endpoints and two reference methods.

After correction, WISER's NDCG@5 and MCC gains over Direct WF Score and MCC gain over H0 with BO remain significant. Table R3.1 in Question 1 reports all six effect sizes, pointwise 95% confidence intervals, and adjusted p-values. We will specify these uncertainty definitions and procedures in Section 4, Appendix B.2, and table notes.

Q1: Paired Inference and Uncertainty Definitions:

Thank you for requesting paired comparisons. We analyze 255 test events, 5,100 valid event–security pairs, and 84 common accounting dates. The “±” values are sample standard deviations across five paired seeds; intervals quantify test-period sampling uncertainty conditional on the fixed trained runs.

We generate 10,000 circular bootstrap resamples using 20-date blocks shared across methods and seeds, retaining complete events and security sets while preserving within-block temporal dependence. Per run, we recompute NDCG@5 by event multiplicity, MCC from the pooled confusion matrix, and Sharpe from daily net returns, then average paired differences across seeds. The Table below reports pointwise 95% intervals from a centered, symmetric, non-studentized bootstrap. Half-widths equal the 95th percentile of ∣Δ∗−Δ∣|\Delta^*-\Delta|, with Δ\Delta the observed mean paired difference and Δ∗\Delta^* its bootstrap replicate. Two-sided p-values use centered tail counts with a plus-one correction; Holm jointly adjusts all six p-values.

Reference methodEndpointWISER − referencePaired 95% CIRaw pHolm-adjusted p
Direct WF ScoreNDCG@5+0.0240[0.0053, 0.0427]0.01060.0424
Direct WF ScoreMCC+0.0271[0.0116, 0.0425]0.00020.0010
Direct WF ScoreSharpe+0.1000[-5.2202, 5.4202]0.97321.0000
H0 with BONDCG@5+0.0200[-0.0007, 0.0408]0.05950.1785
H0 with BOMCC+0.0231[0.0136, 0.0325]0.00010.0006
H0 with BOSharpe+0.5800[-4.7927, 5.9527]0.83511.0000

After Holm correction, significant gains at 0.05 remain for NDCG@5 and MCC versus Direct WF Score and MCC versus H0 with BO. Both mean Sharpe gains have intervals spanning zero. These results support incremental gains in entity ranking and directional prediction under matched training budgets. We add procedures and statistics to Appendices B.2 and C.1.

Q2: Portfolio Rules and Metric Rankings

Thank you for highlighting the differing rankings. Appendix B.2 defines entity relevance using reliability-weighted absolute abnormal responses ∣z∣|z|, so large negative reactions can improve NDCG@5; MCC covers all three directional classes. The long-only portfolio buys up to five securities with the highest-confidence positive predictions, holds them for 20 local trading days, retains unused cash, and charges 10 basis points per side.

Table 7 reports NDCG@5 gains of 0.004 and 0.010 for H0 with BO and default WFE over Direct WF Score, despite lower Sharpe. The Table below varies only holdings per event or transaction costs, keeping predictions, execution conventions, holding periods, and allocations across events and markets fixed. Within each event, all methods use identical weighting and cash rules.

Portfolio ruleWISERDirect WF ScoreH0 with BOWFE default
Top-5; 10 bp; main2.882.782.302.43
Top-5; 0 bp3.023.012.452.60
Top-3; 10 bp2.762.802.252.36
Top-10; 10 bp2.792.682.322.46

Direct WF Score exceeds H0 with BO and default WFE in Sharpe across all four settings. WISER achieves the highest mean Sharpe in three, with margins over Direct WF Score ranging from -0.04 to +0.11. NDCG@5 and MCC remain unchanged because predictions are fixed. Table 9 separately reports entailment 7.2 percentage points higher and unsupported mechanism rates 9.8 points lower for WISER's selected training responses relative to Direct WF Score. Together with Question 1's paired task comparisons, these results support improved supervision quality and future task performance.

Q3 Timestamp Manifest and Memory Probes:

Thank you for recommending these checks to strengthen auditability. Section 3.2 specifies candidate locking before outcome access, while Eq. (1) and Appendix B.3 require test outputs before scored returns. We added memory probes for 60 historical development events across China, the United States, Europe, and Japan, and will release a machine-readable manifest covering evidence provenance, generation settings, candidate commitments, test timestamps, and probe records, with publication and replay execution times recorded separately.

We test 15 events per market using the frozen generator in Appendix B.2, disabled retrieval, fixed decoding settings, and independent contexts, totaling 180 calls. Each target is a later-disclosed fact absent from and not directly inferable from event-time evidence. A retains real identities; B anonymizes identities and removes date/location cues; both withhold the target. C supplies it as a reading control. A hit requires all prespecified numerical fields, units, and semantic roles to match after formatting normalization. Probe outputs are excluded from candidate pools and student training.

ConditionHits / eventsHit rate95% CI
A: Real identities; target withheld2/603.33%[0.41%, 11.53%]
B: Anonymized identities; target withheld0/600.00%[0.00%, 5.96%]
C: Target supplied (reading control)57/6095.00%[86.08%, 98.96%]
RecordRecorded fieldsAudit check
EvidenceEvent/run IDs; evidence versions, hashes, sources, availability times; cutoff tet_eEvidence available by tet_e
GenerationModel, retrieval, and prompt versions; decoding settings; seedsReproduce generation conditions
Candidate commitmentCandidate hashes; lock and first-outcome-read timestampsLock precedes outcome access
Test predictionSubmission time TeoutputT_e^{\mathrm{output}}; scoring start aea_ete≤Teoutput<aet_e \le T_e^{\mathrm{output}} < a_e
Memory probeCondition timestamps; targets, sources, publication times; matching and anonymization rules; prompts; raw outputsReproduce A/B/C checks

A, B, and C achieve target-reproduction rates of 3.33% (2/60), 0.00% (0/60), and 95.00% (57/60), respectively (Table R3.3). C shows the highest recovery, while B has no observed matches. With the generator frozen and retrieval disabled, this pattern is consistent with strong extraction from supplied evidence and limited reproduction of withheld targets under the tested prompts. Together, the probes and timestamp records strengthen WISER’s contribution of auditable curation using historical feedback followed by prospective student inference using event-time inputs alone. We also provide an illustrative example of the timestamp records in Table 10.

Reviewer_V6WMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WISER, a method that uses historical market outcomes to select synthetic data for supervised fine-tuning (SFT) of large language models. It targets two core challenges: the model-internal curation loop, in which the generator and the selector share model priors and lack external verification against actual outcomes, and the delayed and noisy nature of external financial market feedback. First, WISER uses a frozen Qwen3.5-9B to generate K = 8 admissible structured candidate responses for each event and locks them before the replay workflow accesses outcome tables. Second, WISER splits market outcomes into an early selection window [d_e, d_e+2], which supplies market compatibility and observation reliability as selector inputs, and a later confirmation window [d_e+6, d_e+20], which supplies candidate preference targets. The World Feedback Energy model (WFE) learns these preferences, and its four hyperparameters are selected through constrained Bayesian optimization. Finally, the selected responses are used to train a student model that receives only event-time information at inference. Evaluation on 255 post-freeze events shows gains of WISER over fixed market-feedback scoring (Direct WF Score), with additional comparisons across markets, temporal blocks and student models.

Critical strengths

1.The motivation is clear, and the paper addresses two real bottlenecks in domain SFT curation: the circularity of model-internal scoring and the noise in external market outcomes. The framework is designed around these challenges, it selects responses for student fine tuning using historical market feedback and its reliability to improve future financial event reasoning. Within the replay workflow, the protocol also guards against outcome leakage: candidates are locked before outcome access, the selection and confirmation windows are separated by a gap, and an embargo separates the temporal partitions. The framework thus cleanly connects retrospective, outcome-guided curation with prospective reasoning that relies on event-time inputs alone.

  1. The experiments are comprehensive and designed to identify the contribution of each component. Beyond comparisons with ten same-pool curation baselines and four financial systems across four global markets (Table 1), the paper runs alignment falsification tests (Figure 3b), showing that shifting dates, permuting tickers or confirmation outcomes, or shuffling evidence reduces the selector NDCG gain from 0.118 to at most 0.011, so the signal is indeed tied to the correct outcomes. It also evaluates robustness across three post-freeze temporal blocks and three student backbones (Table 2), and ablates key components (Table 3) to confirm their contributions. Together, these results provide evidence for several of the main design choices.
Critical weaknesses

1.The method-specific gains over Direct WF Score are relatively modest compared with the gains from introducing market feedback. Direct WF Score improves over Uniform-1 by 0.120 in NDCG@5, 0.140 in MCC and 1.22 in Sharpe (Table 8), whereas WISER adds only 0.024, 0.027 and 0.10 on the same metrics. This suggests that much of the overall improvement comes from using external market feedback, while the incremental value of the learned selector and constrained configuration search is comparatively modest. Although Appendix C.1 compares WFE with a contextual linear selector, it does not establish whether the proposed energy-based parameterization offers advantages over conventional nonlinear alternatives. Proposition E.4 and Theorem E.6 establish the ability to model reliability-dependent interactions and ranking reversals, but these properties are not unique to WFE. Since the free-energy formulation in Equation 5 is closely related to a context-conditioned softplus network, comparisons with a standard MLP or GBDT ranker using the same features, training targets, and tuning budget would help establish whether WFE provides distinctive empirical benefits. Given the relatively small incremental gains over Direct WF Score, their statistical reliability and the justification for the additional methodological complexity are particularly important.

2.The pipeline contains many engineering choices and hand-set constraints whose selection and effects are not sufficiently justified, including the temporal feedback windows, the seven reliability components, and the four feasibility thresholds. Although the paper provides component-level ablations and some robustness analyses, it does not systematically examine sensitivity to these specific design choices. Since these settings may affect both the availability and quality of market-feedback supervision, it remains unclear how robust the reported improvements are to alternative reasonable specifications. Further justification or targeted sensitivity analyses would strengthen the empirical conclusions.

3.The relationship between market-outcome alignment and the quality of financial reasoning remains insufficiently established. The paper acknowledges that realized market reactions are not semantic or causal ground truth, as they may reflect market expectations, concurrent events, and other external influences. Although the reliability-conditioning mechanism and alignment falsification experiments help address noise in market feedback, the primary evaluation metrics focus on entity ranking, directional prediction, and investment performance. These metrics do not necessarily establish that the selected responses provide more factually accurate explanations or better financial reasoning. A complementary evaluation of the factual accuracy and reasoning quality of selected responses, potentially using a small expert-annotated sample, would help establish whether WISER improves reasoning quality beyond alignment with subsequent market movements.

Questions for authors

1.Could you compare WFE with other nonlinear rankers, such as a standard MLP or GBDT, or with a linear selector that includes interaction terms, under the same features, preference targets and tuning budget? Alternatively, could you explain what property of the WFE parameterization is expected to improve selection beyond the nonlinear interactions already established by Theorem E.6?

2.Section 4 states that event-paired bootstrap intervals and Holm-adjusted primary comparisons are reported. However, the confidence intervals or adjusted p-values for the incremental improvements of WISER over Direct WF Score and H0 with BO are not clearly presented. Could the authors explicitly report these pairwise comparisons for all three primary endpoints to establish which incremental improvements are statistically supported?

3.Could you explain how the temporal window offsets, reliability components, and feasibility thresholds were selected, or provide sensitivity analyses showing whether the main conclusions remain robust under alternative reasonable settings?

4.To what extent does WISER improve the factual accuracy and reasoning quality of the selected responses independently of their alignment with subsequent market movements? Could the authors provide an expert-annotated evaluation of reasoning quality to complement the market-based performance metrics?

4 comments

We are grateful for the reviewer’s careful assessment and helpful suggestions regarding WFE’s incremental contribution, statistical reliability, configuration sensitivity, and reasoning quality. In response, we provide matched nonlinear and interaction baselines, paired tests with Holm correction, reliability ablations, window and threshold sensitivity analyses, and blinded expert evaluation of factual grounding and reasoning coherence.

W1: Learned Selector Gains and Complexity:

Thank you for asking us to clarify learned selection’s incremental value. Tables 7 and 8 in Appendix C.1 compare methods under shared candidate pools and student training budgets. WISER improves Student NDCG@5, MCC, and Sharpe over Direct WF Score by 0.024, 0.027, and 0.10, respectively. Against H0 with the same 64 BO evaluations, the gains are 0.020, 0.023, and 0.58.

As described in Sections 3.3 and 3.4, WFE learns interactions among candidate attributes, early compatibility, and reliability from temporally separated confirmation preferences. Under shared default settings, WFE improves the three student metrics over H0 by 0.022, 0.025, and 0.40. Adding BO yields further gains of 0.014, 0.016, and 0.45. These comparisons quantify gains from learned interactions and configuration search.

Additional computation occurs during offline curation. Equation 5 analytically marginalizes binary latent variables, avoiding enumeration of hidden states. Given features and context biases, scoring costs O(Kd(H+1))O(Kd(H+1)), with BO capped at 64 evaluations. Students use only event-time inputs at inference. Questions 1 and 2 report conventional nonlinear comparisons and paired statistical analyses, respectively.

W2: Reliability Components and Configuration Sensitivity:

Thank you for asking us to evaluate reliability components. Appendix B.2 defines seven checks of observation adequacy, price quality, and event confounding. Table R2.1 compares the full configuration with seven ablations, assessing their individual contributions to supervision selection.

Each ablation removes one component from input construction and fitting. We recompute security reliability over the remaining six components as r−j=(∏c≠jsc)1/6r_{-j}=\left(\prod_{c\ne j}s_c\right)^{1/6}, where scs_c denotes component cc, and update compatibility in both windows, fitting confirmation preferences, and supervision weights. Raw validity rules and reference evaluations for BO, validation, and testing remain fixed.

Removed componentStudent NDCG@5MCCSharpe
None, full configuration0.6300.3352.88
Estimation coverage0.6290.3342.87
Tradable-day coverage0.6240.3282.79
Corporate-action checks0.6300.3352.88
Concurrent events0.6210.3242.73
Price consistency0.6250.3292.81
External comovement0.6200.3222.71
Residual stability0.6230.3272.77

Full WISER matches or exceeds every ablation. Removing external comovement reduces Student NDCG@5, MCC, and Sharpe by 0.010, 0.013, and 0.17; removing concurrent events yields declines of 0.009, 0.011, and 0.15. The largest declines support downweighting feedback associated with broad market movements and concurrent disclosures. Question 3 examines alternative feedback windows and the four feasibility thresholds.

W3: Market Alignment and Explanation Quality:

Thank you for asking us to assess explanation quality. Sections 3.2 to 3.4 and Appendix B.2 describe how WFE combines internal quality ratings, completeness, and citation validity with early compatibility and reliability. The internal ratings assess task fulfillment, evidence consistency, and claim support without subsequent market data.

Table 9 in Appendix C.2 audits selected responses for 120 events. Against Direct WF Score, WISER raises evidence entailment from 74.4% to 81.6% and reduces unsupported mechanisms from 31.8% to 22.0%, improvements of 7.2 and 9.8 percentage points. WISER’s preference rate over Direct WF Score is 72.2%, with a 95% CI of 65.2% to 78.2%.

Replacing WISER’s rationales lowers entailment to 41.6% and raises unsupported mechanisms to 74.5%. Original responses are preferred in 87.8% of comparisons, with a 95% CI of 82.6% to 93.3%. These results support better grounded supervision under shared candidate pools and demonstrate the audit’s sensitivity to explanation quality. Question 4 adds expert evaluation with subsequent market data withheld.

Q1: Matched Nonlinear Selector Comparisons

Thank you for suggesting stronger selectors. We add a softplus MLP with one hidden layer, learnable output coefficients, and a separately fitted context dependent linear skip matching WFE’s form. The interaction selector adds ρb\rho b, ρm\rho m, and vec⁡(bm⊤)\operatorname{vec}(bm^{\top}), where bb contains six internal features, mm three early compatibility features, and ρ\rho early reliability. These 27 terms expand 20 inputs to 47 using the same underlying information.

All models share candidates, confirmation preferences, event weights, and evaluation references. Each uses the listwise objective in Equation 6. MLP and WFE widths range over 8,16,32,64\\{8,16,32,64\\}. Each class receives 64 configuration evaluations and 1,000 Adam updates per configuration. Students use five paired seeds and 259,600 effective supervision tokens. Table R2.2 reports selector diagnostics and student means across seeds.

SelectorSelector NDCGStudent NDCG@5MCCSharpe
H0 + BO0.7020.6100.3122.30
Explicit interaction selector0.7260.6200.3232.68
MLP0.7360.6260.3302.81
WFE / WISER0.7420.6300.3352.88

WFE leads all four metrics, exceeding MLP by 0.006, 0.004, 0.005, and 0.07, respectively. For fixed context and parameters, define S=−FS=-F and th=ηh(g)+wh⊤ϕt_h=\eta_h(g)+w_h^{\top}\phi. Equation 5 gives

∇ϕ2S=∑h=1Hσ(th)[1−σ(th)]whwh⊤⪰0,\nabla_{\phi}^{2}S=\sum_{h=1}^{H}\sigma(t_h)\left[1-\sigma(t_h)\right]w_h w_h^{\top}\succeq 0,

where σ\sigma is the sigmoid function. WFE’s unit output coefficients make the score convex in ϕ\phi while preserving reliability interactions; the MLP learns these coefficients. Fixing WFE’s output coefficients restricts nonlinear combinations during preference fitting. The matched gains support this constrained parameterization. Appendices C.1 and E.4 will include the comparisons and structural analysis.

Q2: Paired Intervals and Holm Correction:

Thank you for requesting explicit statistical evidence. The “±” values denote sample standard deviations across five paired seeds. The Table below separately estimates test-period uncertainty and applies Holm correction to six comparisons across three endpoints and two references.

We use 255 events, 5,100 valid event security pairs, and 84 common portfolio accounting dates. Holding trained runs fixed, we generate 10,000 circular bootstrap resamples with blocks of 20 accounting dates shared across methods and seeds, retaining complete event security sets. Within each run, we recompute NDCG@5 by event multiplicity, MCC from pooled confusion matrices, and Sharpe from daily returns, then average paired endpoint differences across five seeds. Pointwise 95% CIs are symmetric and non-studentized, with half-widths given by the 95th percentile of absolute errors ∣Δb∗−Δ^∣|\Delta_b^*-\widehat{\Delta}|, where Δ^\widehat{\Delta} is the observed paired difference and Δb∗\Delta_b^* its bootstrap replicate. Two-sided pp-values use the plus-one tail-count correction.

Reference methodEndpointMean differencePaired 95% CIRaw ppHolm-adjusted pp
Direct WF ScoreNDCG@5+0.0240[0.0053, 0.0427]0.01060.0424
Direct WF ScoreMCC+0.0271[0.0116, 0.0425]0.00020.0010
Direct WF ScoreSharpe+0.1000[-5.2202, 5.4202]0.97321.0000
H0 with BONDCG@5+0.0200[-0.0007, 0.0408]0.05950.1785
H0 with BOMCC+0.0231[0.0136, 0.0325]0.00010.0006
H0 with BOSharpe+0.5800[-4.7927, 5.9527]0.83511.0000

WISER’s six mean gains are positive, with MCC against both references and NDCG@5 against Direct WF Score surviving Holm correction. MCC against H0 with BO has the lowest adjusted p of 0.0006, while both Sharpe intervals span zero. WFE learns confirmation preferences through nonlinear interactions that let reliability reshape candidate rankings. Under matched pools and training budgets, these gains demonstrate the value of reliability conditioned energy selection for future reasoning.

Q3: Window and Feasibility Threshold Sensitivity:

Thank you for asking us to justify these settings. Section 3.3 uses [0,2][0,2] for immediate reactions and [6,20][6,20] for confirmation, separated by [3,5][3,5]. Equation 7 constrains weighted coverage, effective sample size, source completeness, and directional share, with thresholds fixed at 0.80, 100, 0.95, and 0.80 before BO access. Table R2.4 tests nearby alternatives.

Window variants preserve early inputs, temporal splits, and the eligible development pool, with confirmation ending by trading day 20. Changed training measurements are recalibrated on the original calibration set and frozen before refitting the selector and training the student. Evaluation references for BO, validation, and all 255 test events remain fixed across variants. Threshold variants change one condition at a time.

The Table below is the Window durations are in trading days and intervals include both endpoints.

SettingStudent NDCG@5MCCSharpe
Default0.6300.3352.88
Gap 1 day; [4,18][4,18], 15 days0.6270.3312.81
Gap 2 days; [5,19][5,19], 15 days0.6310.3342.86
Confirmation 5 days; [6,10][6,10]; gap 3 days0.6220.3252.74
Confirmation 10 days; [6,15][6,15]; gap 3 days0.6260.3302.81
Weighted coverage ≥ 0.750.6280.3322.84
Weighted coverage ≥ 0.850.6290.3342.86
ESS ≥ 800.6270.3312.83
ESS ≥ 1200.6310.3362.89
Source completeness ≥ 0.900.6260.3292.80
Source completeness ≥ 0.980.6300.3352.87
Directional share ≤ 0.750.6280.3332.84
Directional share ≤ 0.850.6270.3302.79

Across threshold variants, NDCG@5, MCC, and Sharpe range from 0.626 to 0.631, 0.329 to 0.336, and 2.79 to 2.89. Every tested setting exceeds Direct WF Score’s 0.606 NDCG@5 and 0.308 MCC. Extending confirmation from 5 to 15 days raises the three metrics from 0.622, 0.325, and 2.74 to 0.630, 0.335, and 2.88. These gains support using extended confirmation information for preference learning. Proposition E.24 gives max⁡eπe≤1/Neff\max_e\pi_e\le1/\sqrt{N_{\mathrm{eff}}}, so ESS at least 100 bounds each normalized fitting weight by 10%. These results complement the component ablations in W2.

Q4: Blinded Expert Evaluation:

Thank you for suggesting expert evaluation. We added a blinded audit of 60 events, 15 per market, comparing WISER, Direct WF Score, and Internal selections from shared candidate pools. Two raters with finance backgrounds scored independently; a third adjudicated disagreements. Raters saw only event-time evidence and responses, with method identities and subsequent market data withheld. Order was randomized, and identical responses shared ratings.

Factual support averages the supported fraction of six claims per response across events. Reasoning scores are 0 for critical contradictions, 1 for a missing transmission link, and 2 for supported premises and coherent transmission; the pass rate is the proportion of responses scoring 2. The unsupported mechanism response rate is the proportion containing at least one unsupported mechanism. The Table below reports percentage-point differences and pointwise 95% confidence intervals from 10,000 bootstrap resamples stratified by market and paired by event, using unrounded results.

MetricWISERDirect WF ScoreInternalWISER − DirectWISER − Internal
Factual support ↑86.1%78.3%70.6%+7.8 [3.6, 11.9]+15.6 [11.1, 20.0]
Reasoning pass rate ↑68.3%50.0%31.7%+18.3 [5.0, 31.7]+36.7 [23.3, 50.0]
Unsupported mechanism responses ↓18.3%31.7%41.7%-13.3 [-26.7, 0.0]-23.3 [-38.3, -8.3]

Against Direct WF Score, WISER improves factual support and reasoning pass rates by 7.8 and 18.3 points, with both intervals above zero. The unsupported mechanism rate falls by 13.3 points. These blinded ratings support better factual grounding and transmission coherence in selected supervision.

Thank you for the detailed response and additional experiments, which support the value of reliability-filtered market feedback for curating financial reasoning data. At the same time, the results also show that the Sharpe improvement over Direct WF Score is not statistically significant and that WFE performs very similarly to a standard MLP, so the authors may consider toning down the corresponding claims in the final revision. Overall, our main concerns have been addressed, and I will maintain our positive score at 3.

Reviewer_uyKAOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how historical market outcomes can guide the curation of synthetic supervision for financial event reasoning. WISER locks candidate responses before accessing outcome tables, constructs early market-compatibility features and later confirmation preferences from nonoverlapping windows, and learns a reliability-conditioned energy selector. Constrained Bayesian optimization configures the selector within 64 evaluations. One selected response per curation event is then used to fine-tune a student that receives only event-time information at inference.

Critical strengths
  1. The paper separates candidate generation, early feedback, later confirmation supervision, response selection, and future student evaluation. Historical outcomes influence the student through selected responses, while inference uses event-time inputs alone. This directly addresses an important source of ambiguity in outcome-guided synthetic-data curation.
  2. The shared candidate pools, matched student training budgets, five paired seeds, and Direct WF Score baseline make the main curation comparison informative. The H=0 controls in Table 7, alignment falsification tests, cross-student transfer, and semantic audit provide additional evidence beyond aggregate performance.
Critical weaknesses
  1. The statistical evidence does not yet establish the reliability of the incremental gains. The text mentions reporting 95% bootstrap intervals and Holm-adjusted comparisons, but the corresponding numerical results are not identifiable; the meaning of "±" in the tables is unclear (seed variability, standard error, or CI), which is critical for evaluating the ~0.10–0.11 Sharpe improvement.
  2. The 255 test events and 20-session holding periods involve overlapping holdings, common shocks, and shared securities, but the bootstrap method fails to explain how this dependence is preserved when estimating portfolio Sharpe uncertainty.
  3. The contribution of the specific WFE formulation is insufficiently isolated. While Equation (5)'s energy-based formulation (WFE) beats linear models, it fails to prove an advantage over conventional nonlinear rankers or simpler selectors with explicit interaction features.
Questions for authors

How does WFE compare with a conventional nonlinear ranker or an explicit interaction-feature selector under the same features, confirmation targets, fitting protocol, and search budget? In the “w/o Reliability” ablation, which components are removed: security-level weights, early reliability inputs, confirmation weights, or reliability filtering? Are evaluation definitions held fixed?

4 comments

We thank the reviewer for raising concerns about statistical uncertainty, portfolio dependence, WFE’s contribution, and reliability ablations. We address these through paired bootstrap inference with Holm correction, block-length sensitivity analyses, matched MLP and interaction controls, and separate ablations of four reliability pathways under fixed evaluation definitions, clarifying the evidence for WISER’s incremental gains.

W1: Statistical Reliability and Uncertainty:

Thank you for requesting clearer statistical evidence. Section 4 and Appendix B.2 report five matched runs (seeds 11, 23, 37, 51, and 71); “±” denotes the sample standard deviation across these runs. We now report test-period sampling uncertainty separately through paired comparisons in Table R1.

We generated 10,000 paired bootstrap resamples using circular blocks of 20 accounting dates, shared across methods and seeds while holding the trained runs fixed. For each endpoint, we computed WISER-minus-reference differences within each seed and averaged them across the five seeds in each resample. Table R1 reports pointwise 95% confidence intervals and two-sided pp-values from a centered, symmetric, non-studentized procedure. Holm correction covers all six tests: three primary endpoints against two reference methods.

Reference methodPrimary endpointMean differencePaired 95% CIRaw ppHolm-adjusted pp
Direct WF ScoreEntity NDCG@5+0.0240[0.0053, 0.0427]0.01060.0424
Direct WF ScoreDirection MCC+0.0271[0.0116, 0.0425]0.00020.0010
Direct WF ScoreCost-adjusted Sharpe+0.1000[-5.2202, 5.4202]0.97321.0000
H0 with BOEntity NDCG@5+0.0200[-0.0007, 0.0408]0.05950.1785
H0 with BODirection MCC+0.0231[0.0136, 0.0325]0.00010.0006
H0 with BOCost-adjusted Sharpe+0.5800[-4.7927, 5.9527]0.83511.0000

WISER achieves positive mean gains across all six comparisons, with three significant after Holm correction. MCC against H0 with BO shows the strongest evidence at adjusted p = 0.0006, followed by MCC and NDCG@5 against Direct WF Score at 0.0010 and 0.0424. WFE learns confirmation preferences through nonlinear interactions among candidate attributes, early market compatibility, and reliability, allowing feedback quality to reshape candidate rankings. These gains demonstrate the value of reliability conditioned energy selection for converting historical feedback into better student supervision and improving future entity ranking and direction prediction from event time inputs alone.

W2: Overlapping Holdings and Common Shocks:

Thank you for highlighting dependence in Sharpe inference. We bootstrap daily net portfolio returns constructed under Appendix B.2, incorporating concurrent events, aggregated positions in shared securities, holding periods of 20 local trading days, and transaction costs. On each common accounting date tt, we form rt=(rt(m,s))m,s\mathbf{r}_t=(r_t^{(m,s)})_{m,s} across methods mm and seeds ss. Sampling identical contiguous circular date blocks preserves contemporaneous joint movements and within-block serial dependence.

Each resample concatenates blocks in their internal date order and truncates the series to its original length. We recompute Sharpe for every run and average the paired differences across five seeds. Corresponding date indices resample events with their complete security sets and fixed labels. NDCG@5 is aggregated by event multiplicity; MCC is recomputed from each run’s pooled confusion matrix. The primary block length is 20 accounting dates. Table R2 compares lengths of 5, 10, 20, and 30 using 10,000 resamples each, with all other settings fixed, and reports pointwise 95% CIs for WISER minus reference Sharpe differences.

Block length (accounting dates)95% CI vs. Direct WF Score95% CI vs. H0 with BO
5[-3.546, 3.746][-3.239, 4.399]
10[-4.487, 4.687][-3.869, 5.029]
20[-5.220, 5.420][-4.793, 5.953]
30[-5.304, 5.504][-4.971, 6.131]

Both Sharpe intervals widen from 5 to 30 accounting dates and consistently include zero. Endpoints at 20 and 30 dates differ by at most 0.178. Portfolio aggregation and synchronous block resampling account for shared exposures and temporal dependence while preserving pairing. This consistency supports the stability of WISER’s financial evaluation under market dependence.

W3: Contribution of the WFE Architecture

Thank you for asking us to isolate the WFE architecture. Equation 5 in Section 3.4 and Appendix E.4 characterize its reliability interactions. Let ρe\rho_e denote event reliability, χek\chi_{ek} the remaining inputs, and SekS_{ek} the negative free energy. Define Δkℓ=Sek−Seℓ\Delta_{k\ell}=S_{ek}-S_{e\ell} and uekj=ηe,j+wχ,j⊤χek+wρ,jρeu_{ekj}=\eta_{e,j}+w_{\chi,j}^{\top}\chi_{ek}+w_{\rho,j}\rho_e. With context, other inputs, and parameters fixed, Propositions E.4 and E.5 give

\frac{\partial \Delta_{k\ell}^{\mathrm{H0}}}{\partial \rho_e}=0,\qquad \frac{\partial \Delta_{k\ell}^{\mathrm{WFE}}}{\partial \rho_e}=\sum_{j=1}^{H}w_{\rho,j}\left[\sigma(u_{ekj})-\sigma(u_{e\ell j})\right].\tag{R1}

Here σ\sigma is the sigmoid function. H0’s shared reliability term cancels from candidate comparisons, while WFE allows reliability responses to vary across candidates. Theorem E.6 constructs a ranking reversal using one hidden unit. Table 7 matches target temperature, reliability threshold, and regularization. WFE improves Selector NDCG, Student NDCG@5, MCC, and Sharpe over H0 by 0.047, 0.022, 0.025, and 0.40 under default settings. With 64 BO evaluations per scorer, the corresponding gains are 0.040, 0.020, 0.023, and 0.58. Both comparisons share candidate pools, student training budgets, and five paired seeds. These gains span both selector quality and student prediction performance.

Figure 3a provides complementary diagnostics stratified by reliability. WFE connects feedback quality with candidate attributes and market compatibility to guide supervision selection. Question 1a further evaluates its specific parameterization against conventional nonlinear rankers and selectors with explicit interactions.

Q1a: Matched Comparisons with Stronger Selectors:

Thank you for suggesting stronger selectors. We add an MLP with one hidden layer and learnable output weights, using the same candidate input with 20 dimensions and the same context. We augment the linear H0 selector with explicit interactions ψ=[ρb;ρm;vec⁡(bm⊤)]\psi=[\rho b;\rho m;\operatorname{vec}(bm^\top)], where bb contains six internal features, mm three early compatibility features, and ρ\rho early reliability. These 27 additional terms yield 47 inputs from the same underlying information.

For this architecture comparison, all scorers share locked candidates, frozen confirmation preferences, event loss weights, fitting support, evaluation references, fixed target temperature, and filtering rules. Each uses the listwise objective in Equation 6, 1,000 Adam updates per configuration, and 64 configuration evaluations. Students share initialization, five paired seeds, and 259,600 effective supervision tokens. The Table reports selector diagnostics and student means across these seeds. This comparison assesses how each architecture uses common information and supervision under matched fitting and configuration evaluation budgets.

SelectorSelector NDCGStudent NDCG@5Direction MCCCost-adjusted Sharpe
H0 + BO0.7020.6100.3122.30
Linear selector with explicit interactions0.7260.6200.3232.68
MLP0.7360.6260.3302.81
WFE / WISER0.7420.6300.3352.88

WFE ranks first on all four metrics, exceeding MLP, the strongest added baseline, by 0.006, 0.004, 0.005, and 0.07, respectively. Both added selectors improve over H0, with WFE achieving further gains under the matched protocol. Its context conditioned linear and softplus terms jointly model candidate attributes, compatibility, and reliability through confirmation supervision. Appendix C.1 will include architectures, capacity ranges, and search settings.

Q1b: Reliability Ablations under Fixed Evaluation:

Thank you for requesting precise ablation definitions. The Table below isolates the four reliability pathways in Sections 3.3 to 3.5; the Table 3 setting “w/o Reliability” removes them jointly.

Security weighting removal assigns unit weights to originally valid securities in both windows, recomputes compatibility and fitting preferences, and retains original event reliability. Early input removal drops reliability and its missingness indicator, reducing 20 inputs to 18. Confirmation weights become normalized uniform weights on unchanged fitting support. Removing the reference threshold of 0.30 retains hard validity checks and excludes tied targets. All configurations share 255 test events, 5,100 valid event security pairs, labels, metric definitions, portfolio rules, and frozen selector evaluation targets and weights.

ConfigurationFit eventsSelector NDCGStudent NDCG@5MCCSharpe
Full3000.7420.6300.3352.88
w/o security weighting3000.7110.6100.3062.35
w/o early reliability input3000.7310.6230.3272.76
Equal confirmation weights3000.7190.6150.3152.51
w/o reliability threshold3370.7390.6280.3322.85
w/o Reliability3370.6840.5990.2922.06

Full WISER leads all four metrics. Joint removal reduces Student NDCG@5, MCC, and Sharpe by 0.031, 0.043, and 0.82, respectively. Security weighting has the largest individual effects, with declines of 0.020, 0.029, and 0.53; threshold removal has the smallest, at 0.002, 0.003, and 0.03. Security weights affect both compatibility and confirmation preferences, while the other pathways govern candidate comparisons, supervision contributions, and fitting support. These results support integrating reliability throughout feedback construction and preference learning. Definitions will be added to the Table 3 notes and Appendices B.2 and C.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.