- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
REATS uses an LLM to assign sample-dependent weights to candidate forecasters. Its inputs combine textual and numerical temporal features with weight priors retrieved from training examples. Rule-generated reasoning supervision supports SFT, followed by forecasting-oriented GRPO. The evaluation covers eight datasets, specialized and foundation-model candidate pools, changes in the candidate set, and transfer settings. The reported improvements are promising. The remaining questions concern which training objective produced the reported results and what LLM reasoning adds over alternatives with the same information.
Critical strengths
-
REATS targets adaptive weighting of candidate forecasts. The method specifies a hybrid temporal representation, a training-only retrieval pool, API-free construction of reasoning supervision, and two-stage training. Using the LLM as an ensemble router gives these components a defined forecasting role.
-
The evaluation covers eight datasets and two classes of candidate forecasters, with candidate-set changes, transfer, and CoT/RAG ablations. Table 2 reports average MSE of 0.1384 versus 0.1597 for the foundation-model OptWval comparison, and 0.1080 versus 0.1352 for the small-model OptWtr comparison. The component ablations support the usefulness of CoT and retrieval within this pipeline. Together, these results make the overall supervised router worth considering.
Critical weaknesses
-
The implemented reward needs to be connected to the reported results. Table 2(b) reports Electricity/Weather MSE of 0.1476/0.0206, whereas Table 8 reports 0.1463/0.0061. Different candidate pools, seeds, or settings could explain the difference, but their correspondence is unspecified. Section 3.4 assumes a nonnegative row-specific MSE gap, although nine reference rows are deliberately nonoptimal. The prior static audit summarized in the supplied revision record reports that the code clips negative gaps to zero. Clipping avoids the previously suspected negative-gap reciprocal singularity. The analysis still needs to account for saturation after an auxiliary reference is beaten. The same audit records percentage-parsing, column-order, and exact-row-count validation issues. Their frequency and effect in the reported runs remain unknown. An immutable source snapshot was unavailable in that reassessment. Please identify the version used and provide representative parsed rollouts and reward records from the reported runs. A parser patch or a possible explanation would not, by itself, connect the implemented objective to the existing results.
-
The experiments support the complete routing pipeline more clearly than the incremental contribution of LLM reasoning. Appendix A.1.3 states that RLMC receives raw histories, whereas REATS receives engineered features and retrieved weight priors. A direct retrieved-weight baseline and a lightweight router with the same features, supervision, and retrieval information would help isolate that contribution. Algorithm 1 explicitly constructs reasoning targets from oracle weights, which is disclosed training supervision. Prediction gains and plausible explanations nevertheless do not establish explanation faithfulness. Candidate transfer also rebuilds retrieval pools from the new candidates' historical training predictions. This historical-calibration requirement should be stated explicitly.
Questions for authors
-
Which candidate pools, seeds, split boundaries, and training configurations produced Table 2(b), Table 8, and Figure 6? Please map these results to their runs. Explain the differing Electricity/Weather entries and how the stated 7:1:2 split corresponds to the reported ETT sample counts.
-
Please identify the reward and output-validation version used in the reported GRPO runs, with its version identifier and configuration. State how it handles gap clipping, percentage parsing, column order, required row count, and inference-time row selection. Include representative parsed rollouts and reward records, and any summaries showing how often the identified cases occurred. If those cases materially affected training, a targeted comparison with corrected behavior would help assess their impact. This concern remains open until the implemented behavior is connected to the reported results.
-
Are there results for direct retrieved weights and a lightweight router with the same candidate predictions, engineered features, training labels, and nearest-neighbor weight information? If absent, these controls would address the contribution-attribution question. Please state the historical calibration required for candidate transfer. A controlled faithfulness test would be needed to support the stronger explanation claim; otherwise, the interpretation should be limited to generated, inspectable rationales.
Additional presentation feedback Figure 2's oracle-weight block lists nonnegative simplex constraints and then gives the inverse-Gram closed form. Section 3.3 states that this closed form applies without the nonnegativity constraint and that the actual oracle uses a constrained quadratic program. Please distinguish these two problems in the figure. For Figures 1–2, please supply a corrected, editable SVG, drawn directly or produced with AI assistance and manually checked. Check the mathematical notation, labels, and signal paths against the method. Empirical plots should be regenerated from their numerical records and plotting code.
6 comments
Response to Critical Weakness 2
Critical Weaknesses 2:
Weakness point (1): The experiments support the complete routing pipeline more clearly than the incremental contribution of LLM reasoning. Appendix A.1.3 states that RLMC receives raw histories, whereas REATS receives engineered features and retrieved weight priors. A direct retrieved-weight baseline and a lightweight router with the same features, supervision, and retrieval information would help isolate that contribution.
Answer: We add the lightweight router baselines, including GBDT, LightGBM, and MLP, which are provided with the same engineered features, supervision, and retrieved weight information as REATS. We also add the direct retrieved-weight baseline with the same training retrieval pool and Euclidean distance as REATS (retrieval top 3 priors).
Results are shown in Table 1, REATS achieves the best overall performance, demonstrating the benefits of its LLM-based design beyond the shared input information, including CoT reasoning, GRPO-based reward optimization, and structured weight generation.
Table 1. Ensemble learning on small model groups. The experimental settings are identical to those used in the main paper. We report per-dataset test MSE for direct retrieval, lightweight routers, and REATS.
| Method | Exchange | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Electricity | Traffic | Avg. ↓ |
|---|---|---|---|---|---|---|---|---|---|
| Direct retrieval Top-1 | 0.2682 | 0.1778 | 0.2017 | 0.1066 | 0.0943 | 0.2853 | 0.1565 | 0.1109 | 0.1752 |
| Direct retrieval Top-3 mean | 0.2268 | 0.1612 | 0.1846 | 0.0853 | 0.0902 | 0.0862 | 0.1559 | 0.1036 | 0.1367 |
| kNN model selection | 0.2117 | 0.1524 | 0.1767 | 0.0908 | 0.0722 | 0.2795 | 0.1464 | 0.0667 | 0.1495 |
| GBDT | 0.2213 | 0.1559 | 0.1805 | 0.0818 | 0.0822 | 0.1138 | 0.1529 | 0.1018 | 0.1363 |
| LightGBM | 0.2213 | 0.1558 | 0.1806 | 0.0817 | 0.0824 | 0.1185 | 0.1527 | 0.1019 | 0.1369 |
| MLP | 0.2333 | 0.1669 | 0.1832 | 0.0836 | 0.0828 | 0.0875 | 0.1531 | 0.1024 | 0.1366 |
| RLMC (raw history) | 0.3875 | 0.1572 | 0.3029 | 0.0970 | 0.0838 | 0.1666 | 0.1549 | 0.0666 | 0.1771 |
| RLMC (hybrid features) | 0.4447 | 0.1629 | 0.1938 | 0.0948 | 0.0821 | 0.1740 | 0.1478 | 0.0666 | 0.1708 |
| REATS-GRPO | 0.1587 | 0.1409 | 0.1700 | 0.0725 | 0.0720 | 0.0206 | 0.1476 | 0.0819 | 0.1080 |
Weakness point (2): Algorithm 1 explicitly constructs reasoning targets from oracle weights, which is disclosed training supervision. Prediction gains and plausible explanations nevertheless do not establish explanation faithfulness. Candidate transfer also rebuilds retrieval pools from the new candidates' historical training predictions. This historical-calibration requirement should be stated explicitly.
Answer: Thanks for the valuable insights. We evaluated rationale–weight consistency using 80 test outputs per model, sampled from 8 datasets (10 per dataset). In both REATS-SFT and REATS-GRPO, the candidate prioritized in the rationale received the highest weight in the first output row in all 80 cases (80/80). The Weather example in the Appendix further illustrates this consistency. While these results provide empirical evidence of rationale–weight consistency, the causal faithfulness of the explanations is still worth further investigation, and we leave the development of faithfulness metrics to future work. Besides, we will clarify in the revised manuscript that candidate transfer requires historical forecasts from the new candidates on the training split to rebuild the retrieval pool, while the router remains fixed.
Response to Questions For Authors
Questions For Authors 1:
Question point (1): Which candidate pools, seeds, split boundaries, and training configurations produced Table 2(b), Table 8, and Figure 6? Please map these results to their runs. Explain the differing Electricity/Weather entries.
Answer:
(1) The difference between the results in Table 2(b) and Table 8 arises from the different loss coefficients used in Eq. (4). Table 2(b) uses the G1 configuration, whereas Table 8 reports the best-performing configuration among G1–G6.
(2) The candidate pools, random seeds, data-split boundaries, and all other training settings are identical across Table 2(b), Table 8, and Figure 6. The only difference is the choice of loss coefficients in Eq. (4).
Please refer to the answer of Weakness point (1) in the response to Critical Weaknesses 1 for details.
Question point (2): how the stated 7:1:2 split corresponds to the reported ETT sample counts.
Answer: By adjusting the split boundaries, we can achieve an approximately 7:1:2 ratio of valid sliding windows across the training, validation, and test sets while preserving temporal order.
Questions For Authors 2:
Please see our response to Critical Weakness 1.
Questions For Authors 3:
Please see our response to Critical Weakness 2.
Finally, we thank the reviewer for the additional presentation suggestions. We will revise Figures 1–2, particularly the mathematical formulations, to clearly distinguish the unconstrained closed-form solution from the simplex-constrained QP used to obtain the actual oracle weights. We will also check the labels and signal paths and provide the editable SVG files in the project repository. In addition, we have verified that all empirical plots are generated from the corresponding numerical records and plotting code.
Response to Critical Weakness 1
Critical Weaknesses 1:
Weakness point (1): The implemented reward needs to be connected to the reported results. Table 2(b) reports Electricity/Weather MSE of 0.1476/0.0206, whereas Table 8 reports 0.1463/0.0061. Different candidate pools, seeds, or settings could explain the difference, but their correspondence is unspecified.
Answer: Table 2(b) uses the default reward coefficients in Eq. (4), G1 with (λ1,λ2,λ3)=(0.8,0,0.2), for all datasets. The “GRPO with Reward Mapping” column in Table 8 reports the lowest observed test MSE for each dataset across the six reward-coefficient settings in Figure 6.
As the values highlighted in red in Figure 6 of the paper, G5 (1,0,0) gives the lowest MSE on Electricity and Weather (0.1463 and 0.0061), compared with 0.1476 and 0.0206 under G1. For Exchange, G1 gives the lowest MSE (0.1587), so both tables report the same value. We will clarify this distinction in the Table 8 caption.
Weakness point (2): Section 3.4 assumes a nonnegative row-specific MSE gap, although nine reference rows are deliberately nonoptimal. The prior static audit summarized in the supplied revision record reports that the code clips negative gaps to zero. Clipping avoids the previously suspected negative-gap reciprocal singularity. The analysis still needs to account for saturation after an auxiliary reference is beaten.
Answer: Thank you for pointing this out. Our implementation clips negative row-wise MSE gaps to zero, so the MSE reward for a row saturates at 1 when the prediction matches or beats its reference. We will make this explicit in Section 3.4. The highest logged aggregate rewards were in the 0.90–0.99 range. Designing a reward that continues to distinguish improvements after a prediction reaches the maximum reward is an interesting direction for future work.
Weakness point (3): The same audit records percentage-parsing, column-order, and exact-row-count validation issues. Their frequency and effect in the reported runs remain unknown. An immutable source snapshot was unavailable in that reassessment. Please identify the version used and provide representative parsed rollouts and reward records from the reported runs. A parser patch or a possible explanation would not, by itself, connect the implemented objective to the existing results.
Answer: Thanks for the valuable advice. We would like to clarify a few points in the hope of addressing your concerns.
(1) The anonymous repository submitted with the paper contains the same code used for all reported experiments.
(2) We implemented a fallback mechanism for parsing failures: when valid weights cannot be recovered, they are replaced with a uniform allocation, which is generally less competitive than the learned adaptive weights. Such cases are extremely rare, accounting for less than 0.5% of samples, and thus have negligible impact on the reported results.
(3) We also examined the rollout-level reward records. The rewards generally increase from approximately 0.2 at the beginning of training to around 0.95 as training progresses.