- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to addressing the efficient usage of synthetic price path data on training time-series foundation models. The authors define the utility gap as the reduction in forecasting risk relative to the same adaptation without replacement. UGapSyn makes the comparison by continuing adaptation from a shared state with synthetic or real windows at the replaced positions, and selects the configuration of a controllable GARCH-jump generator using the gap measured on validation data. Experiments on US and China stock markets further show the effectiveness of UGapSyn.
Critical strengths
- High-quality financial data is usually limited. Effective usage of synthetic data is important to train financial time series foundation model.
- A controllable GARCH-jump generator is reasonable design with finance insights.
- The flow of this paper is easy to follow.
Critical weaknesses
- As real history market data is always considered very valuable, I do not think budget is the bottleneck for training financial foundation model. As a result, I think adding synthetic data as extra information is better than replacing them with real historical data.
- The setting of this work has the potential to be more general. Besides financial settings, it may be extend to the training of different time-series foundation models on different domains.
- Even though there are lots of generation models working on creating synthetic financial market data, there is no gold standard to verify the how realistic they are. Unlike CV or NLP, it is hard to distinguish the quality of synthetic financial time-series data. Addressing the quality of synthetic financial data is the preliminary for the success of this work.
- UGapSyn use the gain of data samples to decide the usefulness of different data samples. However, explanability is quite important in finance, using one performance on some specific testing periods is not promising enough. It is more likely to be overfitting than real improvement.
- Very large scale experiments are required as this is about training foundation models. The current datasets are at most medium-level for me.
- Using time-series data sample with part of overlap can enlarge the size of available data. Maybe this can be used in the design of this framework.
- Although I can see the potential of this work, the current proposed method is a bit incremental for me.
Questions for authors
See weakness above.
6 comments
W2: Generality beyond finance
We focused the experiments on finance because market history is limited and non-stationary, so the question of whether synthetic data are worth using is most pressing there. The method itself is not tied to finance, for two reasons.
(1) The definition and the measurement are not specific to finance. Replacement utility only requires a pretrained model, a real training sequence and held-out feedback data (Section 2), and the three backbones we adapt are general time-series foundation models.
(2) The generator is the only finance-specific part. In another domain GARCH-jump would be replaced by a generator suited to the data there. Replacing it with fractional Brownian motion or a block bootstrap leaves the rest of the method unchanged, and the measurement proceeds in the same way (Appendix B.8, Table 14).
Revision: Section 5 now notes that the definitions make no financial assumption and that the approach can be applied to other domains with a suitable generator.
W3: Realism of synthetic financial data
The quality of generated financial series is indeed hard to judge directly, and there is no accepted standard of realism. This is the situation the paper addresses. It measures the effect of synthetic data on forecasting directly, so a realism standard is not a prerequisite for it, and the introduction now says so. The three points below describe this measurement, why fidelity metrics cannot take its place, and the basic checks that generated paths still pass.
(1) Judging synthetic financial data without a gold standard for realism. Without such a standard, we judge synthetic data by replacement utility. From the same starting state and under the same budget, adaptation continues with either the synthetic windows or the real windows they replace, and the two forecasting errors are compared (Section 2). Synthetic data count as good only if they help forecasting more than the real data they replace.
(2) Whether fidelity metrics could serve as the quality criterion. Fidelity metrics do not rank sources consistently in our experiments. On the US panels smaller fidelity distances generally go with larger utility, but on CSI300 their rank correlations with utility lie between -0.35 and +0.17 (Section 4.3, Tables 6 and 8). Fidelity can also miss a harmful source: on US_71 the energy distance of SDForger is in the same range as that of our parametric sources, yet its MSE is 0.2192 against 0.1877 for real-data adaptation (Table 1).
(3) Basic quality checks on the generated paths. Generated prices must be finite and positive, and a path must not duplicate a real window or another generated path (Section 3.1). Table 6 reports fidelity scores of all sources together with their utility, and Table 8 relates all six scores to utility.
Revision: (1) The introduction now states that financial series have no gold standard of realism against which fidelity scores can be calibrated. (2) It also states that evaluation against the displaced real data requires no separate judgment of realism.
W4: Explainability and overfitting
Thank you for raising these two concerns. Addressing them made the paper more explicit about how each decision is made and why the selected data help. Usefulness is decided on validation data, and test data are not used in selecting the configuration or in adjusting the policy (point (1)). The gain in the main results rests on this single choice, and points (2) and (3) explain why it is unlikely to overfit and what makes the selected data useful.
(1) Whether usefulness is decided by performance on the test periods. Usefulness is decided on the validation periods, and the test periods serve only to evaluate the frozen configuration. Each panel is split in time order into training, calibration, validation and test periods (Table 17). The generator configuration is selected on validation and then frozen, and runtime adjustments use only the calibration panels, which come before the validation period.
(2) Whether the improvement is overfitting to specific test periods. The selection leaves little room for overfitting. It chooses among a few candidates registered in advance, and the single selected configuration is applied to the three panels and all three backbones without reselection (Appendix B.3). With this one configuration, UGapSyn has lower mean test MSE than real-data adaptation in all nine combinations of backbone and panel (Table 1), and the ten-seed paired intervals on Chronos-Bolt-Base exclude zero on both US panels (Table 2). A separate ten-seed comparison on Chronos-Bolt-Small gives the fixed configuration a lower mean on all three panels as well (Table 4). On a later US period, from January 2024 to September 2026, which played no part in any selection, UGapSyn still has the lower mean (Table 3(b)). The gain is also not confined to one stretch of time: split into half-year blocks, the validated configuration is better than real-data adaptation in 14 of the 20 blocks (see our reply to Reviewer taeP, W3 & Q3).
(3) Explaining why the selected data are useful. The generator is a parametric financial model, so the selected data can be described and their effect traced to specific properties. The selected configuration has zero simulated drift, equal weights on four regions of variance scale and persistence, and 50% replacement. Removing the simulated drift changes test MSE from 0.1855 to 0.1832 on US_71 and from 1.9911 to 1.9531 on CSI300, while on US_14L the two drift settings are nearly equal (1.0874 and 1.0885, Table 3(a)). Where the drift setting matters, paths without a drift estimated from the training period help more, which agrees with a common observation in finance that the average return of one period is a poor guide to the next. The region weights we compared change test MSE by at most 0.0004 in the ten-seed paired comparison (Table 12) and by a few thousandths in the three-seed diagnostic (Table 6). The amount of synthetic data matters as well: lowering the replacement level from 50% to 30% raises test MSE from 0.1829 to 0.1854 on US_71 and from 1.0859 to 1.0990 on US_14L (Table 15).
Revision: (1) Sections 2 and 3 now state that all decisions use only training and feedback data and that test data are used in neither selection nor revision, and Section 4.1 describes the test and later evaluations as held-out. (2) Section 5 notes that, on US_71 and CSI300, the drift setting changes test MSE far more than the region weights compared in Appendix B.8. (3) Appendix B.3 describes the selected configuration in financial terms and states that it is used for all three backbones without reselection.
Thank you for your review and comments! We have considered each of them carefully, and the paper is clearer and better supported as a result. We respond to each point below. All changes in the revised paper are in blue, and the margin tags that refer to your comments are in purple (kb3S·W1, for example). The tags for the other two reviewers are in red and green.
W1: Replacing or adding synthetic data
Thank you for this comment on the problem setting, which led to one of the most useful additions in this revision. The setting does not assume that the computing budget is a bottleneck. It fixes the budget so that synthetic data are compared with the real data they replace under the same conditions (points (1) and (2)). We also ran the experiment you propose, in which synthetic data are added to the real data (point (3)).
(1) Why the budget is fixed even though it is not the bottleneck. We fix the budget so that a change in forecasting error can be attributed to the data, which Section 2 now states. Spending more updates on the real data did not help in our controls either: extending real-data adaptation to 1,092 updates raises test MSE from 0.1877 to 0.2433 on US_71, from 1.1229 to 1.4204 on US_14L and from 1.9575 to 2.0425 on CSI300 (Table 5). The question is therefore which windows should fill a short schedule.
(2) How the method treats the value of real historical data. Real data are the reference in our formulation. A synthetic window counts as useful only if the adapted model forecasts better with it than with the real window it replaces (Eq. 3), and the synthetic configuration is selected by this comparison on validation, where it improves on real-data adaptation (Appendix B.8).
(3) Adding synthetic data as extra information versus replacing real data. Synthetic data help when they are added as well. We kept all 128 real windows in each block and added 64 windows, either synthetic or further real ones, so that both cases use 76 updates.
Adding synthetic windows gives lower MSE than adding real windows by 0.0058 and 0.0174, with intervals excluding zero, and a lower mean than real-data adaptation with 52 updates. The benefit of synthetic data thus does not depend on the replacement setting. Adding more real windows does not improve on real-data adaptation (0.1900 against 0.1876, and 1.1197 against 1.1171), which is consistent with point (1), and replacement with 52 updates still has the lowest mean on both panels. The references in this table are those of Appendix B.8 (Table 10).
Revision: (1) Section 2 now explains that the budget is fixed so that a change in forecasting risk can be attributed to the training data. (2) Section 4.2 summarizes the experiment in which windows are added, with full results in Appendix B.8 (Table 13).