- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes using delayed market outcomes, instead of model-internal scores, to curate synthetic fine-tuning data for financial event reasoning. A frozen generator produces several candidate analyses per event, and a learned energy-based selector, trained to predict later market-confirmed preferences from early market signals and weighted by an event reliability score, picks one candidate for fine-tuning a student model. On a post-freeze test period across several markets, the authors report that the resulting student outperforms text-only data-selection methods, a fixed early-feedback rule, and financial baselines, and they add a theoretical appendix.
Critical strengths
- Leakage-aware, prospective protocol. Candidates are committed before outcomes are seen, windows are disjoint, the test is post-freeze, and pools and budgets are matched, so selection rules are compared cleanly.
- Outcome-based selection clearly helps. A simple early-market rule beats all text-only selectors, consistently across markets, periods and student models, though from a single test window.
Critical weaknesses
- The added complexity is not justified. A fixed early-feedback rule delivers most of the gain, learned variants without the full combination fall below it, and the selector's held-out loss barely beats a uniform predictor. The heavy notation and very long theory appendix, mostly standard identities, are not tied to any empirical claim and obscure the core idea. I hope it can be more concise and reuse symbols.
- Statistical support is missing. The promised paired bootstrap and multiple-testing corrections are absent and the reported spread seems to reflect seeds only, so small margins on one test window are not shown to be significant; this can be added from existing outputs.
Questions for authors
- Please report event-paired intervals and adjusted tests for the learned selector versus the fixed rule, and state what the reported spread measures.
- The fixed rule has lower ranking quality but much higher Sharpe than the learned variants; is this a portfolio-rule artifact rather than better reasoning?
- Can you provide timestamped manifests and a check for memorized post-event knowledge in the generator?
3 comments
Q1: Paired Inference and Uncertainty Definitions:
Thank you for requesting paired comparisons. We analyze 255 test events, 5,100 valid event–security pairs, and 84 common accounting dates. The “±” values are sample standard deviations across five paired seeds; intervals quantify test-period sampling uncertainty conditional on the fixed trained runs.
We generate 10,000 circular bootstrap resamples using 20-date blocks shared across methods and seeds, retaining complete events and security sets while preserving within-block temporal dependence. Per run, we recompute NDCG@5 by event multiplicity, MCC from the pooled confusion matrix, and Sharpe from daily net returns, then average paired differences across seeds. The Table below reports pointwise 95% intervals from a centered, symmetric, non-studentized bootstrap. Half-widths equal the 95th percentile of , with the observed mean paired difference and its bootstrap replicate. Two-sided p-values use centered tail counts with a plus-one correction; Holm jointly adjusts all six p-values.
| Reference method | Endpoint | WISER − reference | Paired 95% CI | Raw p | Holm-adjusted p |
|---|---|---|---|---|---|
| Direct WF Score | NDCG@5 | +0.0240 | [0.0053, 0.0427] | 0.0106 | 0.0424 |
| Direct WF Score | MCC | +0.0271 | [0.0116, 0.0425] | 0.0002 | 0.0010 |
| Direct WF Score | Sharpe | +0.1000 | [-5.2202, 5.4202] | 0.9732 | 1.0000 |
| H0 with BO | NDCG@5 | +0.0200 | [-0.0007, 0.0408] | 0.0595 | 0.1785 |
| H0 with BO | MCC | +0.0231 | [0.0136, 0.0325] | 0.0001 | 0.0006 |
| H0 with BO | Sharpe | +0.5800 | [-4.7927, 5.9527] | 0.8351 | 1.0000 |
After Holm correction, significant gains at 0.05 remain for NDCG@5 and MCC versus Direct WF Score and MCC versus H0 with BO. Both mean Sharpe gains have intervals spanning zero. These results support incremental gains in entity ranking and directional prediction under matched training budgets. We add procedures and statistics to Appendices B.2 and C.1.
Q2: Portfolio Rules and Metric Rankings
Thank you for highlighting the differing rankings. Appendix B.2 defines entity relevance using reliability-weighted absolute abnormal responses , so large negative reactions can improve NDCG@5; MCC covers all three directional classes. The long-only portfolio buys up to five securities with the highest-confidence positive predictions, holds them for 20 local trading days, retains unused cash, and charges 10 basis points per side.
Table 7 reports NDCG@5 gains of 0.004 and 0.010 for H0 with BO and default WFE over Direct WF Score, despite lower Sharpe. The Table below varies only holdings per event or transaction costs, keeping predictions, execution conventions, holding periods, and allocations across events and markets fixed. Within each event, all methods use identical weighting and cash rules.
| Portfolio rule | WISER | Direct WF Score | H0 with BO | WFE default |
|---|---|---|---|---|
| Top-5; 10 bp; main | 2.88 | 2.78 | 2.30 | 2.43 |
| Top-5; 0 bp | 3.02 | 3.01 | 2.45 | 2.60 |
| Top-3; 10 bp | 2.76 | 2.80 | 2.25 | 2.36 |
| Top-10; 10 bp | 2.79 | 2.68 | 2.32 | 2.46 |
Direct WF Score exceeds H0 with BO and default WFE in Sharpe across all four settings. WISER achieves the highest mean Sharpe in three, with margins over Direct WF Score ranging from -0.04 to +0.11. NDCG@5 and MCC remain unchanged because predictions are fixed. Table 9 separately reports entailment 7.2 percentage points higher and unsupported mechanism rates 9.8 points lower for WISER's selected training responses relative to Direct WF Score. Together with Question 1's paired task comparisons, these results support improved supervision quality and future task performance.
Q3 Timestamp Manifest and Memory Probes:
Thank you for recommending these checks to strengthen auditability. Section 3.2 specifies candidate locking before outcome access, while Eq. (1) and Appendix B.3 require test outputs before scored returns. We added memory probes for 60 historical development events across China, the United States, Europe, and Japan, and will release a machine-readable manifest covering evidence provenance, generation settings, candidate commitments, test timestamps, and probe records, with publication and replay execution times recorded separately.
We test 15 events per market using the frozen generator in Appendix B.2, disabled retrieval, fixed decoding settings, and independent contexts, totaling 180 calls. Each target is a later-disclosed fact absent from and not directly inferable from event-time evidence. A retains real identities; B anonymizes identities and removes date/location cues; both withhold the target. C supplies it as a reading control. A hit requires all prespecified numerical fields, units, and semantic roles to match after formatting normalization. Probe outputs are excluded from candidate pools and student training.
| Condition | Hits / events | Hit rate | 95% CI |
|---|---|---|---|
| A: Real identities; target withheld | 2/60 | 3.33% | [0.41%, 11.53%] |
| B: Anonymized identities; target withheld | 0/60 | 0.00% | [0.00%, 5.96%] |
| C: Target supplied (reading control) | 57/60 | 95.00% | [86.08%, 98.96%] |
| Record | Recorded fields | Audit check |
|---|---|---|
| Evidence | Event/run IDs; evidence versions, hashes, sources, availability times; cutoff | Evidence available by |
| Generation | Model, retrieval, and prompt versions; decoding settings; seeds | Reproduce generation conditions |
| Candidate commitment | Candidate hashes; lock and first-outcome-read timestamps | Lock precedes outcome access |
| Test prediction | Submission time ; scoring start | |
| Memory probe | Condition timestamps; targets, sources, publication times; matching and anonymization rules; prompts; raw outputs | Reproduce A/B/C checks |
A, B, and C achieve target-reproduction rates of 3.33% (2/60), 0.00% (0/60), and 95.00% (57/60), respectively (Table R3.3). C shows the highest recovery, while B has no observed matches. With the generator frozen and retrieval disabled, this pattern is consistent with strong extraction from supplied evidence and limited reproduction of withheld targets under the tested prompts. Together, the probes and timestamp records strengthen WISER’s contribution of auditable curation using historical feedback followed by prospective student inference using event-time inputs alone. We also provide an illustrative example of the timestamp records in Table 10.
We sincerely appreciate the reviewer’s thoughtful comments and constructive suggestions on methodological clarity, statistical evidence, portfolio evaluation, and auditability. Guided by this feedback, we clarify loss interpretation and theoretical connections, provide paired statistical analyses and portfolio sensitivity checks, and add generator memory probes and specifications for timestamped audit records to improve overall transparency.
W1 Selection Gains and Theoretical Clarity:
Thank you for asking about complexity and theoretical clarity. Under matched candidate pools and student budgets, WISER improves NDCG@5, MCC, and Sharpe by 0.024, 0.027, and 0.10 over Direct WF Score, and by 0.020, 0.023, and 0.58 over H0 under equal BO budgets of 64 evaluations, as reported in Tables 7 and 8. H0 with BO and default WFE also improve NDCG@5 and MCC over the fixed rule. Additional fitting and search occur offline. Equation 5 analytically marginalizes latent variables, giving scoring cost O(Kd(H+1)) given features and context biases. Student inference uses only event-time inputs.
Figure 4a reports a held-out CN diagnostic of 2.075, compared with uniform cross-entropy log8≈2.079. Proposition E.11 decomposes per-event cross-entropy as
ℓe=H(Qe)+KL!(Qe∥pθ(⋅∣e)).
Target entropy determines the maximum improvement over uniform cross-entropy. Equation 6 adds regularization. We will report cross-entropy, target entropy, and regularization separately under identical events and weights. Figure 4a illustrates optimization; Table 7 evaluates selection quality and student outcomes.
Table 12 links Theorem E.6's reliability interactions to Table 7, Corollary E.21's confirmation-utility regret bound on fixed events and weights to Appendix C.1, and Theorem E.17's margin stability to Figure 4b. We will streamline Section 3.4 and Appendix E around these connections, consolidate standard derivations and definitions, retain assumptions, and consistently use ρe, Qe, and pθ(⋅∣e) for event reliability, confirmation preferences, and selection probabilities. Questions 1 and 2 address paired statistics and portfolio construction.
W2 Statistical Support for Incremental Gains:
Thank you for requesting explicit statistical support. The “±” values denote sample standard deviations across five paired seeds. With trained runs fixed, bootstrap resamples share date blocks across methods and seeds to estimate test-period uncertainty in paired differences. Holm correction covers six tests across three primary endpoints and two reference methods.
After correction, WISER's NDCG@5 and MCC gains over Direct WF Score and MCC gain over H0 with BO remain significant. Table R3.1 in Question 1 reports all six effect sizes, pointwise 95% confidence intervals, and adjusted p-values. We will specify these uncertainty definitions and procedures in Section 4, Appendix B.2, and table notes.