- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MarketPop, a framework for calibrating executable heterogeneous trader populations to empirical market statistics. It represents endowments, asset allocations, strategy weights, and behavioral parameters through a structured low-dimensional parameterization, and introduces Progressive Hierarchical Inference to calibrate them. Experiments in an equity-market simulator compare calibration methods and population-level ablations using five stylized facts, with evaluation on a short subsequent continuation period.
Critical strengths
The framework connects population calibration to executable trading behavior through a structured parameterization. Comparisons with direct parameter search and joint search in the structured space show improved calibration, supporting the practical value of the proposed procedure within the tested simulator.
Critical weaknesses
-
The incremental benefit of the full population hierarchy is unclear. In Table 2, endowment-only calibration achieves a held-out discrepancy of 4.85, compared with 4.89 for the full method. Although the full method improves calibration fit, this comparison does not establish a generalization benefit from the additional levels.
-
Validation is limited to a five-day calibration horizon and three subsequent continuation days. This provides limited evidence that the calibrated population remains effective across different periods or independent simulation runs.
3.Don't talk about some related works like finevo.
Questions for authors
- What evidence supports the contribution of the higher population levels beyond improved calibration fit, given the endowment-only result in Table 2?
- Was the selected population evaluated with fresh simulation seeds independently of those used during calibration? If so, please report the resulting variability and comparison with endowment-only calibration.
4 comments
Q2: Independent seeds, run variability, and the comparison with -only
Yes. After calibration, we froze the selected configuration and ran 60 trading days of held-out evaluation with ten new seeds unused in calibration. No further calibration occurs during testing, and test results play no role in configuration selection. The table reports all ten runs of Full in the January window, together with the five Facts, the five dynamics metrics, and the held-out Loss.
Table R3-Q2: Ten independent held-out runs of Full.
| Run | ↓ | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | -0.7180 | -0.5127 | 0.4409 | -0.5546 | -0.1303 | -0.5713 | -0.4666 | -0.0293 | -0.4030 | -0.3078 | 5.0827 |
| 2 | -0.6819 | -0.5195 | 0.2798 | -0.7721 | -0.1303 | 0.3560 | -0.1907 | -0.2959 | 0.2180 | 0.2108 | 4.9028 |
| 3 | -0.7207 | -0.4936 | -0.4803 | -1.0212 | -0.1303 | 0.4010 | -0.4474 | 0.2348 | 0.2754 | -0.4934 | 3.6900 |
| 4 | -0.6364 | -0.5375 | 0.0487 | -0.6224 | -0.1303 | -0.4846 | -0.5489 | -0.5636 | 0.7934 | 0.8928 | 2.9319 |
| 5 | -0.7538 | -0.5075 | -0.6247 | -0.4629 | -0.1303 | -0.5549 | -0.0479 | 0.7758 | -0.2265 | -0.6633 | 2.2908 |
| 6 | -0.6667 | -0.5157 | 0.1008 | -0.4333 | -0.1303 | -0.4711 | -0.8302 | 0.5960 | -0.6321 | -0.6564 | 2.5802 |
| 7 | -0.6324 | -0.5146 | -0.1350 | -0.7130 | -0.1303 | 0.0173 | 0.0445 | -0.2780 | 0.4009 | 0.1471 | 2.5961 |
| 8 | -0.4773 | -0.5099 | -0.2164 | -0.8553 | -0.1303 | 0.2004 | 0.7649 | -0.1698 | 0.3250 | 0.3470 | 2.8211 |
| 9 | -0.6703 | -0.5352 | -0.2500 | -0.6684 | -0.1303 | -0.1371 | 0.1702 | 0.4520 | 0.1100 | 0.0809 | 2.3540 |
| 10 | -0.5971 | -0.5163 | -0.4139 | -0.7113 | -0.1303 | 0.3146 | 0.7963 | -0.1064 | -0.2974 | 0.0525 | 2.2117 |
| Mean | -0.6555 | -0.5162 | -0.1250 | -0.6814 | -0.1303 | -0.0930 | -0.0756 | 0.0616 | 0.0564 | -0.0390 | 3.1461 |
| Min | -0.7538 | -0.5375 | -0.6247 | -1.0212 | -0.1303 | -0.5713 | -0.8302 | -0.5636 | -0.6321 | -0.6633 | 2.2117 |
| Max | -0.4773 | -0.4936 | 0.4409 | -0.4333 | -0.1303 | 0.4010 | 0.7963 | 0.7758 | 0.7934 | 0.8928 | 5.0827 |
On these calibration-independent paths, the five Facts lie essentially within one standard deviation of their reference means, with only isolated runs slightly outside. The five dynamics metrics show more pronounced variation across runs, and some deviations change sign; nevertheless, all lie within one reference standard deviation in every run, although they were not part of the calibration objective. Held-out loss ranges from 2.2117 to 5.0827, with the first two runs relatively high and the rest closer together. These results indicate some repeatability of the selected configuration's held-out performance alongside path-to-path variability; we report the ten-run average together with the per-run range so that the evaluation does not hinge on any single run. (Min/Max are computed per column and do not correspond to any single run.)
For the comparison with -only, we included it as a baseline in the updated main experiments. Under the common setup and budget, endowment-only calibration still exhibits extensive price-limit pinning and a much poorer calibration fit than Full; the calibration-period results are given in Table 1 of the general statement (Comment 5). The ten independent tests above show Full's subsequent performance and run-to-run variability after calibration.
Weakness: short validation window and cross-period performance
We ran the same calibration procedure separately in two windows starting in January and July 2024, each extended to 10 trading days of calibration followed by 60 of held-out evaluation:
- January window: Jan 2–Apr 18, 2024 — calibration Jan 2–15, held-out Jan 16–Apr 18.
- July window: Jul 1–Oct 15, 2024 — calibration Jul 1–12, held-out Jul 15–Oct 15.
These spans count only open trading days on the Shanghai and Shenzhen markets, excluding weekends and holiday closures. Each window searches and selects its configuration on its own calibration period; the configuration is then frozen, with no further calibration during the held-out period.
Table R3-W: Calibration error and held-out performance in the two historical windows.
| Window | ↓ | ↓ | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Jan 2024 | -0.6555 | -0.5162 | -0.1250 | -0.6814 | -0.1303 | -0.0930 | -0.0756 | 0.0616 | 0.0564 | -0.0390 | 1.8474 | 3.1461 |
| Jul 2024 | -0.5621 | -0.4258 | -0.1877 | -0.7014 | -0.1303 | 0.1091 | -0.0357 | 0.1182 | -0.1758 | -0.3875 | 1.7611 | 3.7535 |
Both windows achieve comparable calibration and held-out performance, with overall errors of similar magnitudes; the five Facts deviate similarly, and the five dynamics metrics all lie within one standard deviation of their reference means. The same procedure, calibrated separately in the two periods, thus achieves a good subsequent market match in both. Together with the independent-seed results in our reply to Q2 above, this round of evaluation both broadens the time coverage and examines the repeatability of a fixed configuration across random paths.
Weakness: related work and the relationship to FinEvo
Thank you for pointing out this omission. We agree that FinEvo should be discussed more explicitly.
FinEvo introduces a closed-loop market framework for evaluating financial trading agents, capturing how their interactions, strategy competition, and capital reallocation shape market outcomes. MarketPop addresses a complementary problem: constructing and calibrating complete heterogeneous trader populations using empirical market evidence. Specifically, MarketPop jointly calibrates resource ownership, asset allocation, strategy composition, and behavioral parameters through structured population representations and hierarchical search. While both works connect participant-level mechanisms with market dynamics, FinEvo focuses on agent evaluation and evolutionary dynamics, whereas MarketPop focuses on the empirical grounding of executable trader populations.
We will add a discussion of FinEvo in the revised related-work section and clarify its relationship to MarketPop, particularly the differences in their research objectives and methodological approaches.
Q1: Role of the additional layers beyond calibration fit
We thank you for the careful reading and constructive suggestions.
After fixing the resource-scale, strategy-execution, and order-blocking issues of the original experiments, we reran the main experiments and the four-layer ablation under the revised common setup, focusing on whether each layer's role persists into subsequent simulation.
The original π1-only comparison is retained as a main-experiment baseline; its updated calibration results, together with Full, are listed in Table 1 of the general statement (Comment 5). The layer ablation was changed from progressively enabling layers to a leave-one-layer-fixed design: from the same initial state, π1, π2, π3, or θ is excluded from calibration in turn, and each variant is compared with Full.
After the 10-day calibration, each selected configuration is frozen and evaluated over 60 held-out days with ten test seeds unused in calibration; we report the averages. "w/o" means the layer remains in the population but is not calibrated; the five Facts and five dynamics metrics all come from the held-out period.
Table R3-Q1: Calibration error and held-out performance of the four-layer ablation.
All four ablations have higher held-out losses than Full, so from the same initial state, disabling any single layer worsens overall subsequent performance. Without π1, deviations in absolute-return autocorrelation and some order-book metrics widen; without π2, the match in volatility and absolute-return autocorrelation worsens and the held-out loss is the highest. Without π3, deviations in volatility, the absolute-return–volume relation, volatility persistence, spread, and drawdown increase; without θ, absolute-return autocorrelation, the absolute-return–volume relation, and the five dynamics metrics all deviate more. The dynamics metrics never entered the calibration objective, so their changes provide evidence of the higher layers' contribution beyond the targeted Facts. In the revised setup, the benefits of the full hierarchy appear in both calibration fit and overall held-out loss, as well as in specific market characteristics on independent test paths.