- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TSCOMP-Agent, a multi-agent LLM workflow built with LangGraph. It reads a forecasting model's paper and code, proposes how to split the model into components, implements those components as modules that follow the slot contracts of an existing component framework (TSCOMP / TSGym, Liang et al. 2026), and admits them to a shared pool after deterministic contract checks, reviewer-guided repair and a global consolidation/repair stage (Sec. 3.2, Alg. 1, App. B). The resulting pool is searched with random screening, pruning and TPE, and the top-5 validation configurations are ensembled (Sec. 3.3, App. E.3). The authors make three claims. (i) The workflow reliably extends the pool. Stage-wise ablations are in Sec. 4.2, and a comparison with "Prompt Only" and "Dynamic Workflow" baselines across three LLM backbones is in Sec. 4.3 and Table 18. (ii) The workflow generalizes to newly introduced models (Table 2). (iii) Reassembled models match or beat 13 reference models reproduced inside the framework and 4 AutoML/automated baselines on 8 LTSF datasets. These results hold up under historical-cutoff and backbone-constrained settings, both tested on ILI only (Tables 3-6). In total, 26 new components were extracted from 30 models (Table 11).
Critical strengths
- The problem is real and well motivated. Component-level analysis of forecasting models (TSCOMP, TimeRecipe, CombinationTS) depends on manually curated component spaces that age quickly. Automating the step of turning a paper and its code into slot-conforming components is a useful, practical goal.
- The engineering design is careful. Deterministic code owns all writes to the shared workspace. Compatibility contracts include rationales for each field (Eq. 8, Table 12). Repairs bind each fix to a specific failure, with no-progress detection. Failure attribution is separated from agent repair (App. D.5). Tokens and wall-clock time are logged precisely. The prompts are documented in App. B, which helps reproducibility.
- The stage-wise ablations are honest about trade-offs and report negative findings. Exhaustive review lowers yield (Table 14). Local memory lowers novelty. Distinctiveness feedback lowers recall (Table 16). Standard deviations over 5 runs are given in the appendix.
Critical weaknesses
-
The reassembly gains are not attributed to the agent. The central claim is that the agent-extracted components enable better reassembled models. But no experiment compares the same search and ensemble pipeline on the seed pool C0 alone with the pipeline on C0 plus the 26 new components. Without this ablation, Tables 3-6 could largely reflect (a) per-dataset/horizon search over a large configuration space and (b) top-5 ensembling, neither of which needs the agent. A report of how many of the selected top-5 configurations use agent-added components would also help. This is the most important missing experiment.
-
The baseline comparisons are not like-for-like, and the "outperform SOTA" claims in the conclusion are overstated. (a) "Ours" is an equal-weight ensemble of 5 configurations chosen from about 200 trials per dataset-horizon cell. The baselines are single models with fixed reproduced configurations (App. E.2-E.3). (b) All trials are trained for at most five epochs with a shared training setup (App. E.3), and the baselines appear to use the same protocol. Several reproduced numbers are well below the values usually reported for these models. For example, Table 3 gives iTransformer Traffic at 0.624 and ECL at 0.227 average MSE, and PatchTST Traffic at 0.569. The comparison is therefore against weakened reproductions. (c) Only one random seed is used (seed 1, App. E.3), and the forecasting tables have no variance. On ILI, the validation-selected model is unstable: H=48 MSE is 4.467 against 1.834 for the oracle (Table 21). The conclusion says reassembly "discovers novel architectures that outperform SOTA baselines". It would be more accurate to say the pipeline beats in-framework reproductions under a short, shared training budget.
Questions for authors
- Can you report Table 3 (or at least a subset of datasets) with the same search and ensemble pipeline run on the seed pool C0 alone? How many of the selected top-5 configurations contain agent-added components?
- How many epochs and how much hyperparameter tuning do the reference baselines receive? Why are reproduced numbers such as iTransformer and PatchTST on Traffic and ECL far from commonly reported values? How do the results change for a single validation-selected configuration (no ensemble), or when baselines get the same ensembling?
- Did you check agreement between the LLM judge (MiniMax-M2.7) and humans on Consistency and Novelty? Were the "ground-truth decompositions" in App. D.2 human-made or LLM-assisted (App. D.4)?
- What evidence supports "outperforms manual curation" and "matches human-expert decomposition quality"?