- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PortfolioRouter, a method for selecting among completed portfolio proposals when routing errors have different economic consequences. The paper separates date-level criticality, defined retrospectively using the realized gap between the best and second-best proposals, from candidate-specific switching value relative to a default selector. The method combines a multi-task routing model, a default-relative value model trained on benefit/harm targets, and a deployment gate calibrated on the preceding quarter.
The evaluation uses 23 U.S. ETFs and 25 China ETFs from 2025 through 2026 H1. Candidate allocations are generated by four language models, and the router is evaluated with quarterly walk-forward splits. The full method reports mean cumulative return / Sharpe of 34.86% / 1.51 in the U.S. and 45.92% / 1.87 in China. The paper also reports fixed-agent controls, component ablations, matched objective and predictor comparisons, and diagnostics on when switching contributes value.
Critical strengths
- The paper identifies a meaningful mismatch between routing accuracy and downstream portfolio value. Treating date-level criticality separately from the value of a particular replacement is a sensible formulation: an incorrect choice on a high-gap date can matter much more than several mistakes on low-gap dates. The switching analysis is consistent with this point; only slightly more than half of the overrides beat General, while the aggregate contribution is still positive.
- The experimental section makes a serious effort to separate routing effects from the quality of the proposal pool. In addition to broad baseline comparisons, the paper includes fixed-agent controls, task-removal ablations, alternative predictors under a common switching framework, and attribution/timing diagnostics. These controls are useful because they go beyond simply reporting a higher terminal return.
Critical weaknesses
- Mixed evidence for consequence-aware supervision. Table 12 shows that Economic Switching Value outperforms Ordinary Correctness in China but not in the U.S. Since the full model also benefits from estimator selection, the independent contribution of consequence-aware supervision remains unclear. A more controlled comparison is needed.
- Concerns about backtest validity. The evaluation assumes same-close execution with zero latency and transaction costs. Moreover, historical proposals generated by recent LLMs may contain future information. Delayed-execution tests, transaction-cost analysis, and stronger point-in-time validation would improve credibility.
- Limited methodological novelty. The framework combines established techniques, including multi-task learning, specialist selection, value estimation, and calibrated switching. Its contributions beyond existing cost-sensitive routing and decision-focused learning methods should be clarified.
- Insufficient explanation of performance gains. Although ablations support the effectiveness of individual components, they do not fully explain the substantial improvements over baselines. A clearer analysis of how critical-date detection, value estimation, and selective switching jointly contribute would strengthen the paper.
- Limited robustness evidence. Experiments cover only two markets over a relatively short period, with three runs per method. The substantial variance in China further raises concerns about stability. Additional evaluation across market periods and agent configurations would strengthen generalizability.
Questions for authors
- In the matched objective study (Table 12), Economic Switching Value improves over correctness-based objectives in China but not in the U.S. Which component explains the gap between these matched neural results and the much stronger full PortfolioRouter result? Can the authors provide a comparison that changes the supervision objective while keeping the selected estimator family and the rest of the switching pipeline fixed?
- Can the authors report the main results with an implementable execution delay and with transaction costs tied to realized turnover (for example, several reasonable per-trade or per-turnover cost levels)? I would like to know whether the gain over General remains after accounting for the extra switching.
- How were the four language-model proposal streams made point-in-time? In particular, what information was supplied to each model and what are the relevant training or knowledge-cutoff dates relative to the 2025-2026 evaluation period? If a fully point-in-time proposal pool is not available, please clarify how the reported results should be interpreted given the possible post-date knowledge noted in the paper.
5 comments
Clarification 2/2
Regarding the source of the switching gains, we decomposed the value branch's replacements into benefits and harms and found that beneficial switches contributed more than harmful switches lost in both markets. This extends the switching analysis in Sec. 5.3 and Appendix D.5.2. Adding the value branch to the base policy, formed by the specialist and forest filter, raises return from 27.80% to 34.86% in the U.S. and from 35.76% to 45.92% in China. About 92 and 96 decisions change per run; 55% and 53% improve on the base action. Positive log-return contributions total 1,361 versus 822 bps of harm in the U.S., and 2,006 versus 1,294 bps in China. Successful replacements contribute more than failed ones lose, explaining the aggregate gain even though only slightly more than half of the switches improve on the base action.
Regarding whether extra switching remains worthwhile after trading delay and costs, we replayed the signals at the next open with turnover-based fees and found positive net gains over General in both markets. We kept the original signals fixed (Sec. 5.1) and charged fees on ETF purchases plus sales after price drift. All policies use the same calendar and carry holdings continuously across quarter boundaries. At 5 bps, PortfolioRouter returns 21.27% versus General's 13.49% in the U.S., and 28.64% versus 22.67% in China: gains of 7.78 and 5.97 percentage points. Sharpe ratios are 0.969 versus 0.683 and 1.223 versus 1.008. PortfolioRouter also exceeds all compared routers and fixed LLM experts within the shared proposal pool at this setting, with lower traded notional than General. The next-open advantage over General persists across 0/1/5/10/20 bps, so the gain remains after charging for the ETF trades required by the selected allocations.
Regarding proposal provenance, we examined the prompt template and generation logs, which document date-filtered inputs and retrospective generation on September 20, 2026. All four generators in Sec. 5.1 received the decision date, ETF universe, current and previous seven sessions of OHLCV, current prices and date-filtered macro observations. All four used the same balanced role, temperature 0.3, and long-only allocations with optional cash. The requested model identifiers in the logs are gpt-5.4, qwen3.6-plus, deepseek-v4-flash and gemini-3.5-flash. The logs record the requested aliases but do not establish exact provider model versions or training cutoffs, so possible post-date model knowledge remains a limitation (Sec. 6). The original fixed-agent controls compare policies on the same cached proposals (Appendix D.3).
Regarding dependence on the LLM proposal pool, we retrained the complete pipeline with four price-only experts and found gains over General in both markets over 2025–2026 H1. The experts use equal weight, 20-session momentum, 20-session inverse volatility and a 60-session trend rule. Each uses inputs available through the signal date. Under the same next-open/5-bps setting, the results are:
| Price-only pool | General return | PortfolioRouter return | Gain (percentage points) |
|---|---|---|---|
| U.S. | 40.35% | 42.82% | +2.47 |
| China | 32.30% | 39.16% | +6.86 |
Both markets retain positive routing gains when every proposal is constructed from historical numerical inputs. We also evaluated the previously unused 2026Q3 quarter, where return differences from General are +0.32 percentage points in the U.S. and −3.84 percentage points in China.
Regarding uncertainty from a shared market history, we used paired block bootstrapping and found positive 95% intervals for the U.S. gain over General at all tested block lengths. The three-seed results share one market history (Appendix C.2). We used 2,000 paired block-bootstrap replicates at lengths of 5/10/20 trading days under the original scoring convention. Both the U.S. return and Sharpe improvements over General have positive intervals at all three lengths.
Regarding China's seed dispersion, we retrained with a 70th-percentile criticality threshold and found a positive routing gain with lower variation across seeds. We held the architecture, seeds and training budgets fixed when examining the dispersion reported in Appendix D.1, Table 5. The retrained variant gives 39.66% ± 1.78%, a gain of 5.33 percentage points over its own General head.
Follow-up on supervision controls and backtest validity
Dear Reviewer qCFa,
Thank you again for your detailed review. We are following up on our November 10 responses, “Clarification 1/2” and “Clarification 2/2.” Your questions motivated several additional experiments. For convenience, we summarize the evidence most directly relevant to Q1–Q3:
-
Contribution of economic supervision beyond estimator selection (Q1). With the quarterly estimator families fixed to those selected by the original pipeline, economic supervision achieves returns of 34.86% versus 32.48% for the strongest correctness objective in the U.S., and 45.78% versus 44.22% in China. Each objective’s numerical gate threshold is calibrated on the preceding quarter. The equal-search-budget comparison also favors economic supervision in both markets. Together, these controls provide evidence that the objective contributes beyond estimator-family selection.
-
Gains under delayed execution and turnover-based costs (Q2). With next-open execution and 5-bps costs on actual ETF purchases and sales, PortfolioRouter retains net return gains over General of 7.78 and 5.97 percentage points in the U.S. and China, respectively, alongside higher Sharpe ratios. Under this setting, it also exceeds all compared routing baselines and fixed LLM experts within the shared proposal pool. Its traded notional is lower than General’s in both markets.
-
Evidence without pretrained LLM proposals (Q3). We retrained the complete pipeline using four price-only experts constructed from numerical observations available through the signal date. Under next-open execution and 5-bps costs, the gains over General are 2.47 and 6.86 percentage points. This provides complementary evidence for the routing mechanism without reliance on pretrained LLM knowledge. Possible post-date knowledge remains a limitation of the original LLM backtests.
Our central methodological contribution is the separation of date-level criticality for specialized learning from candidate-specific, default-relative switching value. The additional controls strengthen the empirical evidence for this formulation.
We would be grateful if you could reconsider your overall assessment in light of these additional results. If any concerns remain decisive for your recommendation, could you please identify them so that we can address them during the remaining discussion period?
Thank you for your time and consideration.
Clarification 1/2
We thank the reviewer for their valuable advice. We would like to address these concerns through controlled comparisons of supervision objectives, delayed-execution and robustness experiments, and further explanation of the method's design, switching gains and proposal-generation process.
Regarding whether estimator selection explains the gains, we ran two supervision controls and found economic value ahead of both correctness objectives in both markets. These controls extend the neural comparison in Appendix D.6.1, Table 12. In the first, every objective receives the same estimator families, search space and budget, with selection confined to prior-quarter validation. In the second, we fix the quarterly estimator family, architecture/training budget, protection weight, history aggregation and gate-score type to those selected by the original economic-value pipeline. Both controls share features, chronological training splits, base decisions and target normalization. Numerical gate thresholds are recalibrated for each objective on the preceding quarter, following Sec. 4.3. Three-seed mean cumulative returns are:
Economic value exceeds both correctness objectives in both markets. The advantage remains when correctness objectives receive the same search budget, and when we compare objectives within the original selected models and switching settings.
The full system selects among estimator families (Appendix A, "Value-model search"), whereas Table 12 uses a dual MLP throughout. It selects ExtraTrees in four U.S. quarters and a direct MLP in one, while China uses dual MLP, random forest and ExtraTrees.
Regarding methodological novelty, our contribution is a portfolio-routing formulation that gives date-level criticality and candidate-specific switching value distinct roles in learning and deployment. The distinction matters because a date can have a large top-two return gap while the default already selects the best proposal. Criticality therefore supervises the detector and specialist (Secs. 3 and 4.1), while the value model learns the benefits and harms of each replacement, with an additional penalty for replacing an optimal reference choice (Sec. 4.2, Eq. (8)). At deployment, the router retains accepted specialist interventions and adds further switches only when predicted value relative to General is positive and the gate calibrated on the preceding quarter is satisfied (Sec. 4.3, Eq. (10)). Together, these targets and rules direct specialized learning toward economically important dates and assess each additional replacement on its own economic value. This design relates to multi-expert deferral [1], cost-sensitive deferral [2] and decision-focused prediction [3] through their focus on expert selection and downstream decision quality (Sec. 2).
References
[1] Rajeev Verma, Daniel Barrejon, and Eric Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 11415–11434. PMLR, 2023.
[2] Jean V. Alves, Diogo Leitão, Sérgio Jesus, Marco O. P. Sampaio, Javier Liébana, Pedro Saleiro, Mário A. T. Figueiredo, and Pedro Bizarro. Cost-sensitive learning to defer to multiple experts with workload constraints. arXiv preprint arXiv:2403.06906, 2024.
[3] Adam N. Elmachtoub and Paul Grigas. Smart “predict, then optimize”. Management Science, 68(1):9–26, 2022.