acceptodds
Desk rejected at ICLR 2027

Auto-Chroma: Automating the Design of Test-Time Time-Series Model Portfolios with Memory-Augmented LLM Agents.

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Time series foundation models (TSFMs) deliver accurate zero-shot forecasts across domains, but their accuracy is tied to model sizes that make inference costly at scale. Chroma offers an alternative paradigm: a portfolio of small specialists, each post-trained from a small TSFM on a subset of the post-training data, matches models orders of magnitude larger with only a few specialists active during inference. However, the portfolio is designed heuristically: the data is partitioned once by sampling frequency or domain, although any subset of it could define a specialist. We present **Auto-Chroma**, an agentic framework that casts portfolio design as a search over specialist configurations and runs the search end to end with memory-augmented large language model (LLM) agents. Rather than guessing from a fixed prompt, the agents reason over a memory of every trial, diagnosing each evaluated portfolio and proposing the next as a hypothesis grounded in the full history of the search. On CB-II and fev-bench-mini, with three univariate and multivariate generalists from 4M to 28M parameters, the portfolios found outperform the frequency heuristic portfolio on test in five of the six settings and achieve competitive performance with respect to large TSFM baselines, and under the same budget Auto-Chroma leads greedy and random search throughout, surpassing the final performance of random search within a fraction of the budget while post-training far fewer specialists. Beyond performance, we analyze both the portfolios the search produces and the search process itself: how fragile the heuristic partitions are, whether the portfolios transfer across benchmarks, whether the validation objective faithfully reflects test performance, how similar the portfolios are across repeated runs and orchestrator LLMs, and how the agents use the memory of past rounds. To our knowledge, this is the first work to investigate portfolio construction as an autonomous process, and the first to treat the search and the portfolios it finds as a subject of study in their own right.

Primary area:
learning on time series and dynamical systems
Keywords:
AutoML, time series foundation models, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sAr5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Auto-Chroma, an agentic framework for automating the construction of portfolios of small time-series forecasting specialists. Starting from a compact pretrained generalist, Auto-Chroma searches over specialist configurations, including the post-training datasets, learning rate, and training duration. An analyst LLM reviews the latest results and prior search history, while a strategist LLM proposes subsequent portfolio configurations. A persistent database stores configurations, scores, routing weights, and agent reports. Numerical specialist training, ensemble calibration, and portfolio evaluation remain outside the LLM components.

The method is evaluated on CB-II and fev-bench-mini using three relatively small forecasting generalists, including both univariate and multivariate models. Auto-Chroma is compared with frequency-based and domain-based specialist portfolios, as well as random and greedy search. The proposed method outperforms the frequency-based portfolio on the reported test metric in five of six model–benchmark settings. The paper additionally examines validation-to-test transfer, portfolio similarity across independent runs and orchestrator LLMs, cross-benchmark transfer, use of prior search history, the contribution of the analyst, and computational cost.

Overall, the paper presents a useful extension of specialist-based forecasting ensembles and provides an unusually broad and candid analysis of the search process. However, the experiments establish more clearly that the overall system can find strong portfolios than that its particular memory-augmented, two-agent architecture is responsible for the improvement.

Critical strengths

1. Useful extension of fixed specialist portfolios

The central idea is well motivated. Rather than treating experts as fixed pretrained forecasters and learning only how to route among them, Auto-Chroma searches over what the specialists learn. The method selects each specialist’s post-training datasets and training hyperparameters, thereby changing the individual experts rather than merely changing their ensemble weights. This is a meaningful extension of common forecasting-ensemble approaches based on immutable pretrained experts. The search problem is also practically relevant because smaller specialist portfolios may provide a better accuracy–efficiency tradeoff than substantially larger general-purpose forecasting models.

2. Encouraging improvements over the main heuristic baseline

The most informative comparisons are against the frequency-based specialist portfolio rather than against the initial generalist alone. On CB-II, test MASE skill increases from 17.56 to 19.09 for Chroma-4M and from 19.35 to 19.98 for Chronos-Bolt-Tiny. These results indicate that searching over specialist construction can outperform a reasonable manually designed portfolio. The evaluation also includes three small generalists and covers both univariate and multivariate settings. Auto-Chroma improves over the frequency-based portfolio in five of the six reported model–benchmark combinations. Although some improvements are small, the overall result provides credible evidence that automated specialist construction is useful.

3. Unusually candid analysis of negative results

A major strength is that the paper reports analyses that do not consistently favor the proposed method. The validation–test analysis identifies a failure of the search objective for Chronos-2-Small on fev-bench-mini, rather than presenting only the successful settings. The analyst ablation similarly shows that removing the analyst performs comparably and sometimes better. The cross-benchmark transfer experiment shows that fixed specialist portfolios transferred between benchmarks, with only the ensemble refitted, produce very similar results. These findings weaken some of the potential architectural claims, but their inclusion substantially improves the scientific quality and credibility of the paper. The paper is more informative because it distinguishes positive headline results from limitations observed during analysis.

4. The search process is treated as an object of study

The paper goes beyond reporting only the best portfolio. It analyzes similarities between portfolios found across runs and orchestrator LLMs, dataset coverage, specialist reuse, references to earlier candidates, validation-to-test behavior, and transfer across benchmarks. This provides a richer picture of how the search behaves and makes the system easier to understand and audit. The persistent record of configurations, numerical outcomes, routing weights, and reports may itself be useful operationally, even if its incremental contribution to forecast accuracy is not yet established.

5. Appropriate separation between LLM reasoning and numerical computation

The LLM components propose and analyze portfolio configurations, but specialist training, routing calibration, and evaluation remain outside the LLM. This is a sensible system design because the language models are used for hypothesis generation and search orchestration rather than numerical forecasting or metric calculation.

Critical weaknesses

1. The robustness of the test improvements is not fully established

Several improvements over the frequency-based portfolio are small. Chronos-2-Small improves by only 0.08 MASE skill points on CB-II. On fev-bench-mini, Chroma-4M and Chronos-Bolt-Tiny gain only 0.28 and 0.16 SQL skill points, respectively, while Chronos-2-Small loses 0.51 SQL skill points.

The larger CB-II improvements for the two smaller generalists are encouraging. However, the aggregate observation that Auto-Chroma wins in five of six settings does not establish that the smaller gains would reliably repeat across independent searches, alternative benchmark samples, or different random seeds.

The appendix reports similarity between independently discovered portfolios, which is useful information about search behavior. However, similarity in portfolio composition does not directly quantify variation in final test performance. Repeated end-to-end searches reporting the distribution of test improvements would provide stronger support for the robustness of the results.

2. The contribution of report-based memory is not demonstrated

Figure 16 and Appendix E.2 count references to earlier candidates in the agent reports. This establishes that the agents use the available history, but it does not show that using this history improves portfolio quality. A more direct experiment would compare the full report-based memory with reduced-memory alternatives under the same search budget. Useful comparisons could include:

  1. Configurations and numerical scores without natural-language reports
  2. Only the most recent search round
  3. A fixed-size history
  4. Summarized rather than complete history
  5. No search history

Without such an ablation, it remains unclear whether persistent report-based memory produces better portfolios or mainly provides a more interpretable record of the search.

3. The separate analyst does not show a consistent accuracy benefit

The analyst ablation raises questions about the proposed two-agent architecture. Removing the analyst increases Chroma-4M CB-II test WQL skill from 35.39 to 37.40, and neither architecture consistently dominates in the remaining settings.

The analyst may still contribute useful diagnostics, auditability, or structured reporting. However, the current experiments do not demonstrate that the analyst improves forecasting accuracy or search efficiency. The paper should either provide stronger evidence for the analyst or narrow its claim, positioning the analyst principally as an interpretability and search-documentation component.

4. The LLM-free baselines are not fully capability-matched

The random and greedy baselines are informative, but they do not cleanly isolate the value of LLM guidance.

Random search restricts specialist subsets to at most six datasets, while Auto-Chroma permits arbitrary nonempty subsets. Auto-Chroma can also use cached specialists, previous results, and per-task routing diagnostics to recombine or modify earlier configurations. The compared methods therefore differ in search space, information access, specialist reuse, and proposal mechanism, rather than differing only in whether an LLM generates the candidate.

A stronger comparison would give a conventional optimizer the same configuration space, specialist cache, numerical feedback, proposal budget, and portfolio-evaluation mechanism. Bayesian optimization, evolutionary search, bandit-based search, or a structured mutation strategy would help determine whether the advantage comes specifically from LLM guidance.

Greedy search also approaches Auto-Chroma late in the Chronos-2-Small budget. This suggests that part of the advantage may arise from broader proposal operators or faster exploration rather than from the full agent architecture.

5. Search costs are not comprehensively matched

The paper reports portfolio-evaluation budgets, numbers of unique specialists, and an LLM API cost of USD 1.96. These are useful measurements, but they do not fully represent the cost of the search.

Different methods may train different numbers of specialists, reuse cached specialists at different rates, or incur different routing-calibration and evaluation costs. Equal numbers of portfolio evaluations therefore do not necessarily correspond to equal total computation. The reported LLM cost also excludes specialist training and routing calibration.

Independent runs matched by accelerator time, wall-clock time, FLOPs, or total measured compute would provide a stronger basis for the efficiency claim.

Questions for authors
  1. Repeated-search uncertainty: What is the distribution of final test performance across independent Auto-Chroma runs? In particular, how often do the smaller improvements over the frequency-based portfolio change sign, and are the reported gains statistically distinguishable from run-to-run variation?

  2. Value of report-based memory:Can the authors compare the full report-based memory against simpler alternatives under the same search budget, such as configurations and numerical outcomes only, recent-round-only context, a fixed-size history, or no history? The current analysis shows that agents reference previous candidates, but not whether this improves portfolio quality.

  3. Role of the analyst: Given that removing the analyst is competitive and improves some reported results, what claim do the authors intend to make about this component? Is its primary contribution forecasting accuracy, search efficiency, interpretability, or auditability?

  4. Capability- and cost-matched non-LLM search: Can the authors compare Auto-Chroma with a conventional optimizer that has the same search space, specialist cache, numerical and per-task feedback, proposal budget, and total computational budget? Please include specialist-training and routing-calibration costs, since equal portfolio evaluations may not imply equal overall compute.

10 comments

We appreciate your careful assessment of our work. We hope that the new experiments and clarifications strengthen your confidence in the paper, and we kindly ask you to consider a higher score. All changes in the revised paper are marked in blue.

W1/Q1: Robustness of the test improvements

Repeated searches. Appendix E.1 compares the portfolios of three independent searches with an identical configuration on fev-bench-mini. We have now evaluated all three on the test split (test skill score against Seasonal Naive, SQL, %; search 1 is the one that the paper reports):

ModelMetricFrequency heuristicSearch 1Search 2Search 3Searches above the heuristic
Chroma-4MSQL35.8836.1636.1036.583 of 3
Chronos-2-SmallSQL49.7549.2449.9250.122 of 3

Auto-Chroma is above the frequency heuristic in 5 of the 6 searches. For Chronos-2-Small, the search reported in the paper is the lowest of the three, and the two other searches are above the frequency heuristic. The search is therefore not underperforming for this model: the reported search fell below the frequency heuristic by chance. In addition, the frequency heuristic itself is sensitive to the composition of the corpus: when we add downsampled data, it loses in all eight validation comparisons and up to 3.90 points of test skill score on CB-II, whereas Auto-Chroma is far less sensitive (RQ1 and Appendix D.1, Figure 8).

Confidence intervals. We computed confidence intervals with the procedure of fev-bench (Shchur et al., 2025): a paired bootstrap over the tasks of a benchmark, with 10,000 resamples. For each model and benchmark, we compare the Auto-Chroma portfolio directly with a heuristic portfolio on the test split, on the primary metric of the benchmark (MASE for CB-II, SQL for fev-bench-mini). The skill score is one minus the geometric mean, over tasks, of the ratio between the error of Auto-Chroma and the error of the heuristic, in percent, so a positive value means that Auto-Chroma is better. Brackets give the 95% confidence interval. Bold marks a positive skill score.

Auto-Chroma against the frequency heuristic:

ModelBenchmarkMetricSkill score, % [95% CI]
Chroma-4MCB-IIMASE1.87 [0.09, 3.73]
Chronos-Bolt-TinyCB-IIMASE0.78 [-0.66, 2.38]
Chronos-2-SmallCB-IIMASE0.10 [-0.85, 0.97]
Chroma-4Mfev-bench-miniSQL0.44 [-1.53, 2.36]
Chronos-Bolt-Tinyfev-bench-miniSQL0.26 [-1.11, 1.75]
Chronos-2-Smallfev-bench-miniSQL-1.02 [-3.20, 1.15]

Auto-Chroma against the domain heuristic:

ModelBenchmarkMetricSkill score, % [95% CI]
Chroma-4MCB-IIMASE3.73 [1.55, 6.02]
Chronos-Bolt-TinyCB-IIMASE4.67 [2.20, 7.34]
Chronos-2-SmallCB-IIMASE2.49 [1.11, 4.07]
Chroma-4Mfev-bench-miniSQL-0.14 [-1.68, 1.67]
Chronos-Bolt-Tinyfev-bench-miniSQL0.31 [-1.23, 1.81]
Chronos-2-Smallfev-bench-miniSQL-1.04 [-2.78, 0.48]

Against the frequency heuristic, the skill score favors Auto-Chroma in 5 of 6 comparisons. The exception is Chronos-2-Small on fev-bench-mini, the case that the repeated searches above cover. Against the domain heuristic, Auto-Chroma is better in all three comparisons on CB-II, by 2.49% to 4.67%. We will add these tables to the appendix of the final camera-ready version.

W2/Q2: Contribution of the report-based memory

We ran two of the comparisons that the reviewer lists, on fev-bench-mini: no search history and configurations and numerical scores without natural-language reports.

(i) Without memory, the strategist is unreliable. In the zero-shot setting, the strategist has no search history: it sees only the two heuristic portfolios and their results, and it makes one proposal. We repeated this with 10 seeds for each model.

Validation skill score, SQL (%), against the frequency heuristic:

ModelZero-shot: minZero-shot: maxZero-shot: meanZero-shot: stdAuto-Chroma
Chroma-4M1.963.942.660.695.41
Chronos-2-Small0.403.001.760.733.67

The proposals vary widely between seeds: for Chronos-2-Small, the standard deviation is about 40% of the mean. Even the best of the 10 proposals stays below Auto-Chroma for both models, and the mean is about half of it. The prior knowledge of the LLM alone is therefore not sufficient, and the memory is necessary.

AuthorsOpenReview ↗

Continue from the first comment

(ii) With the raw history alone, a minimal strategist is already effective. In the second setting, the strategist runs the full 40 rounds with minimal instructions. It knows only the composition and the performance of the past portfolios: it writes no reports, there is no analyst, and it has no access to the ensemble weights. The LLM, the budget, and the benchmark are the same as in the paper. The table reports the best validation skill score (SQL, %, against the frequency heuristic) of each search.

ModelMinimal strategistAuto-Chroma
Chroma-4M3.545.41
Chronos-2-Small4.203.67

For Chronos-2-Small, the minimal strategist is on par with the Auto-Chroma loop. For Chroma-4M, it ends lower than the Auto-Chroma loop. The reasoning of the LLM over the history can therefore be effective even without reports, detailed instructions, or ensemble weights.

Together with the comparison with greedy and evolutionary search (W4 below), these results separate the two components. Greedy and evolutionary search also retain past configurations and numerical results, but they use them through fixed rules, and they end below Auto-Chroma. The zero-shot strategist reasons without memory, and it is unreliable. The memory is therefore necessary, and the reasoning of the LLM over it is effective even in its minimal form. This is plausible: the reports are an explicit form of the reasoning of the LLM, and without them the strategist can still analyze the past portfolios and their results by itself in every round. The reports serve two other purposes. First, they make the search interpretable: each report records why a portfolio was proposed and whether its hypothesis held. Second, they save tokens: the patterns found in earlier rounds persist in the reports, so the strategist does not deduce them again from the raw results in each round. This agrees with RQ6, where removing the analyst gives comparable accuracy. We will state this role of the reports more clearly in the final camera-ready version.

W3/Q3: Role of the analyst

We agree with the reviewer. The paper already states this role of the analyst in RQ6 (Section 4.6, Table 1), which we moved from the appendix to the main text of the revised paper. RQ6 reports that removing the analyst gives comparable accuracy, and that we retain the analyst because its reports make each round interpretable: every report verifies the hypotheses of the preceding strategist report against the measured outcome and records the result. The primary contribution of the analyst is therefore interpretability and auditability, and we do not claim that it improves forecasting accuracy. Its reports also have the practical role that we describe in W2: they persist in the memory, so the agents in later rounds read the verified outcome of each earlier design and do not spend tokens to deduce it again from the raw results. We will make this point clearer in the final camera-ready version.

W4/Q4: Search space and information of the LLM-free baselines

The LLM-free strategies differ from Auto-Chroma only in the proposal step: the budget of 40 candidates, the post-training and evaluation code, the cache of trained specialists, the objective, and the acceptance rule are identical.

Search space of random search. Random search draws each specialist from at most six datasets on purpose. The alternative is a uniform draw over all non-empty subsets of the 44 datasets. Such a draw includes each dataset with probability 1/21/2, so the size of the subset concentrates around 22 datasets, and a subset of at most six datasets has a probability of about 5×10−75 \times 10^{-7}. This search would post-train only specialists on about half of the corpus, which are close to the generalist, and it would not test which division of the corpus is useful. The limit of six datasets is therefore the prior that makes random search a meaningful baseline, and not a restriction that favors Auto-Chroma. Random search also post-trains far more unique specialists (290) than Auto-Chroma, greedy search, and evolutionary search (49 to 57), so it is the most costly of the four strategies (see the updated Figure 4 and Figure 10).

Greedy and evolutionary search are not restricted. Both start from the heuristic portfolios, whose specialists have no such limit (the frequency heuristic divides the 44 datasets among 5 specialists, the largest with 16 datasets), and their operators add, move, and merge datasets without a limit on the size of a subset (Appendix D.2). They can therefore reach any non-empty subset, as Auto-Chroma can.

AuthorsOpenReview ↗

Continue from the second comment

W4/Q4 (continued): Comparison with Bayesian optimization and evolutionary search

Bayesian optimization (BO). BO repeats two steps. First, it fits a predictive model of the score to the candidates evaluated so far. Second, it selects the next candidate as the one that this model rates as most promising. This works well for hyperparameter tuning, because the input is a fixed list of named hyperparameters, the search space is small, and the most promising candidate can be found by sampling. Portfolio search has none of these properties. A portfolio is a set of specialists of variable size, each with its own dataset subset and training hyperparameters, and BO over such spaces is still an open problem (COMBO, Oh et al., 2019; set kernels, Buathong et al., 2020). A BO method for our task would have to solve three problems:

  • (i) Compare two portfolios. The predictive model needs a measure of similarity between candidates. A portfolio has between 1 and 10 specialists, their order has no meaning, and each specialist combines a discrete choice (a subset of the datasets) with continuous ones (learning rate and step count). No standard similarity measure covers this input.
  • (ii) Predict a score that the specialists produce together. The score of a portfolio is not the sum of the scores of its specialists. Ensemble selection fits the weights per task, so a specialist that receives a large weight in one portfolio can receive zero weight in another portfolio that already covers the same tasks. BO observes only one score S(P)S(P) per portfolio. To predict an unseen portfolio, the model must therefore infer from these scores how much each specialist contributed, and in combination with which other specialists.
  • (iii) Find the next candidate. The space is too large for sampling. The corpus has 44 datasets, so one specialist chooses among 244−1≈1.8×10132^{44}-1 \approx 1.8 \times 10^{13} non-empty dataset subsets. A portfolio of 10 specialists chooses 10 distinct subsets, which gives (244−110)≈10126\binom{2^{44}-1}{10} \approx 10^{126} portfolios, before the learning rate and the step count of each specialist are chosen. The space is also discrete, so the most promising candidate cannot be found with gradients or by enumeration.

Set kernels address (i), but (ii) and (iii) remain non-trivial. We therefore see no off-the-shelf BO baseline for this problem; building one would be a research contribution of its own.

Evolutionary search. Evolutionary search is a direct generalization of our greedy baseline, so we added it as a steady-state (μ+1)(\mu+1) evolutionary algorithm: a population of at most μ=5\mu = 5 portfolios starts from the two heuristic portfolios; in every round, parents are picked by tournaments of size 2, two parents are recombined by a uniform crossover over their specialists with probability 0.50.5, and the child receives one mutation of greedy search. Everything else is matched with Auto-Chroma, greedy, and random search: the budget of 40 candidates, the post-training and evaluation code, the objective, and the acceptance rule. The pseudocode is the new Algorithm 7 in the appendix, and the new curves are in the updated Figure 4 (Chroma-4M) and Figure 10 (Chronos-2-Small).

The table reports, on fev-bench-mini, the best validation skill score (SQL, against the frequency heuristic) after 40 candidates and the number of unique specialists that each method post-trains. As in the right panel of Figure 4, the count includes the 11 specialists of the two heuristic portfolios.

ModelMethodBest validation skill score, SQL (%)Unique specialists
Chroma-4MAuto-Chroma5.4157
Chroma-4MGreedy3.0549
Chroma-4MEvolutionary2.4053
Chroma-4MRandom3.57290
Chronos-2-SmallAuto-Chroma3.6755
Chronos-2-SmallGreedy3.5351
Chronos-2-SmallEvolutionary2.2254
Chronos-2-SmallRandom2.64290

Evolutionary search ends below Auto-Chroma for both models, at a comparable number of post-trained specialists, so the comparison is also matched in training compute. It also ends below greedy and random search: every candidate mutates or recombines members of the current population, and the poor portfolios of the initial population leave only one at a time, so its running best rises slowly (Section 4, RQ2).

W5/Q4: Search cost

Training. The cost of a portfolio depends on the number of its specialists, and a cached specialist is not trained again. The number of unique specialists is therefore the measure of the training compute of a search, and the table in W4 gives it for every method. Auto-Chroma, greedy search, and evolutionary search have comparable training budgets, while the training budget of random search is significantly higher.

Reviewer_cDGqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Auto-Chroma, an agentic framework for automatically constructing specialist portfolios for post-training time series generalist models. Auto-Chroma formulates portfolio design as a search problem and employs memory-augmented LLM agents to iteratively propose and analyze specialist configurations based on prior search history. Experiments on multiple benchmarks show that Auto-Chroma outperforms heuristic portfolio construction methods and achieves competitive performance relative to much larger TSFMs.

Critical strengths
  1. Formulating portfolio construction as a search problem is well-motivated, and using memory-augmented LLM agents for the search is a reasonable design choice.
  2. The paper provides many interesting and practical insights. In addition to showing the effectiveness and efficiency of the proposed method, it analyzes the generalization of validation performance to test set and investigates how heuristic partition rules may become fragile as additional data is introduced. These findings could benefit the broader research community.
  3. The paper is clear and easy to follow, and the figures are informative.
Critical weaknesses
  1. The improvement of Auto-Chroma over the heuristic approach on the test benchmarks is small. From Figure 6, the improvements of Auto over Frequency for most models on the two benchmarks are smaller than 1% (the only exception is Chroma-4M on CB-II), and for Chronos-2-small, Auto performs worse than Frequency. Based on the analysis in E.1, even the same LLM has fluctuations in specialist selection across repeated runs. It is thus unclear whether the observed improvements are significant or within variance. Adding confidence intervals would help interpret the results and validate the effectiveness of Auto-Chroma.

  2. For RQ1 and RQ2, performance is mainly compared on the validation set. However, Auto-Chroma directly optimizes the validation skill score, so it is expected to achieve better performance than heuristic or other search strategies when evaluated on the validation set. Could the small improvements on the test split be due to Auto-Chroma potentially overfitting to the validation set?

  3. It is unclear how the individual components of Auto-Chroma contribute. In the ablation studies in E.3, it seems that removing the analyst gives comparable or sometimes even better scores in several cells of Table 4. In addition, portfolios searched on other benchmarks achieve better results in five of the eight comparisons despite Auto-Chroma directly optimizing validation performance on the target benchmark. These observations raise questions about which components are actually driving the gains. Also, it would be helpful to move at least part of the ablation results to the main paper, as they are important for understanding the effectiveness of Auto-Chroma.

  4. The paper compares Auto-Chroma against two non-agentic search baselines: random and greedy. How would Auto-Chroma compare against more advanced search strategies, such as evolutionary search or other LLM-based search methods? Or are existing LLM-based search methods complementary and can potentially be combined with Auto-Chroma? A discussion on this or stronger search baselines would help further validate the effectiveness of Auto-Chroma.

Questions for authors

Please refer to weaknesses.

7 comments

We thank you for your detailed review. If the new results and clarifications below resolve your concerns, we would be grateful if you would consider raising your score. All changes in the revised paper are marked in blue.

W1: Size of the test improvements and confidence intervals

Confidence intervals. As the reviewer suggests, we computed confidence intervals with the procedure of fev-bench (Shchur et al., 2025): a paired bootstrap over the tasks of a benchmark, with 10,000 resamples. For each model and benchmark, we compare the Auto-Chroma portfolio directly with a heuristic portfolio on the test split, on the primary metric of the benchmark (MASE for CB-II, SQL for fev-bench-mini). The skill score is one minus the geometric mean, over tasks, of the ratio between the error of Auto-Chroma and the error of the heuristic, in percent, so a positive value means that Auto-Chroma is better. Brackets give the 95% confidence interval. Bold marks a positive skill score.

Auto-Chroma against the frequency heuristic:

ModelBenchmarkMetricSkill score, % [95% CI]
Chroma-4MCB-IIMASE1.87 [0.09, 3.73]
Chronos-Bolt-TinyCB-IIMASE0.78 [-0.66, 2.38]
Chronos-2-SmallCB-IIMASE0.10 [-0.85, 0.97]
Chroma-4Mfev-bench-miniSQL0.44 [-1.53, 2.36]
Chronos-Bolt-Tinyfev-bench-miniSQL0.26 [-1.11, 1.75]
Chronos-2-Smallfev-bench-miniSQL-1.02 [-3.20, 1.15]

Auto-Chroma against the domain heuristic:

ModelBenchmarkMetricSkill score, % [95% CI]
Chroma-4MCB-IIMASE3.73 [1.55, 6.02]
Chronos-Bolt-TinyCB-IIMASE4.67 [2.20, 7.34]
Chronos-2-SmallCB-IIMASE2.49 [1.11, 4.07]
Chroma-4Mfev-bench-miniSQL-0.14 [-1.68, 1.67]
Chronos-Bolt-Tinyfev-bench-miniSQL0.31 [-1.23, 1.81]
Chronos-2-Smallfev-bench-miniSQL-1.04 [-2.78, 0.48]

Against the frequency heuristic, the skill score favors Auto-Chroma in 5 of 6 comparisons. The exception is Chronos-2-Small on fev-bench-mini, which we discuss next. Against the domain heuristic, Auto-Chroma is better in all three comparisons on CB-II, by 2.49% to 4.67%. We will add these tables to the appendix of the final camera-ready version.

Repeated searches. Appendix E.1 compares the portfolios of three searches with an identical configuration on fev-bench-mini. We have now evaluated all three on the test split (test skill score against Seasonal Naive, SQL, %; search 1 is the one that the paper reports):

ModelMetricFrequency heuristicSearch 1Search 2Search 3Searches above the heuristic
Chroma-4MSQL35.8836.1636.1036.583 of 3
Chronos-2-SmallSQL49.7549.2449.9250.122 of 3

Auto-Chroma is above the frequency heuristic in 5 of the 6 searches. For Chronos-2-Small, the case that the reviewer names, the search reported in the paper is the lowest of the three, and the two other searches are above the frequency heuristic. The search is therefore not underperforming for this model: the reported search fell below the frequency heuristic by chance. In addition, the frequency heuristic itself is sensitive to the composition of the corpus: when we add downsampled data, it loses in all eight validation comparisons and up to 3.90 points of test skill score on CB-II, whereas Auto-Chroma is far less sensitive (RQ1 and Appendix D.1, Figure 8).

W2: Overfitting to the validation set

Our position, as in RQ5 of the paper, is that the small test improvements come from a weak validation signal and not from a search that overfits. Tasks that score few forecasts give noisy validation scores, so validation and test performance correlate poorly on them. Two results support this position.

(i) The portfolios transfer across benchmarks (RQ7, Table 2, reproduced here). We evaluate the best portfolio of each search on the benchmark that it was not searched on, with only the ensemble weights refit. The search never scored this portfolio on the target benchmark, so a portfolio that was overfit to its own validation set would lose here.

Test skill score against Seasonal Naive (%), higher is better:

ModelPortfolioCB-II MASECB-II WQLfev-bench-mini SQLfev-bench-mini MASE
Chroma-4MSearched on target19.0935.3936.1623.16
Chroma-4MTransferred from other18.5536.0436.6223.66
Chronos-2-SmallSearched on target25.3242.1949.2438.63
Chronos-2-SmallTransferred from other25.1642.0949.3938.78

The transferred portfolio is on par with the portfolio searched on the target. The portfolios found by the search are therefore not overfit to the validation set, since they generalize to a new benchmark without loss.

AuthorsOpenReview ↗

Continue from the first comment

(ii) The gap appears without any search. To separate the effect of the search from the effect of the ensemble, we need portfolios that no search selected. Random search provides them: it draws the number of specialists KK and their configurations without looking at any score. We take its 40 portfolios for Chronos-2-Small on fev-bench-mini and compute, for each one, the gap between its validation and its test skill score (validation minus test, in points). A positive gap means that the portfolio looks better on validation than on test. To observe the effect of the number of forecasts on the validation signal, we divide the 20 tasks into the same three groups as the paper uses for this benchmark (Appendix D.5), from the fewest scored forecasts to the most, and report the gap for each KK. If the search caused the gap, these portfolios would show none.

Gap in points, SQL (mean ± standard deviation over nn portfolios):

KKnn8 tasks, 20–160 forecasts6 tasks, 240–420 forecasts6 tasks, 700–11150 forecasts
48-2.79 ± 1.61-0.18 ± 1.67-1.22 ± 0.48
51-2.053.331.21
650.38 ± 2.623.15 ± 1.31-0.62 ± 1.01
79-0.92 ± 1.222.48 ± 2.31-0.27 ± 0.54
860.66 ± 1.754.40 ± 1.950.70 ± 1.22
951.01 ± 3.085.14 ± 3.130.36 ± 1.14
1061.45 ± 0.974.81 ± 0.83-0.17 ± 0.47

Gap in points, MASE:

KKnn8 tasks, 20–160 forecasts6 tasks, 240–420 forecasts6 tasks, 700–11150 forecasts
48-3.28 ± 1.86-0.12 ± 1.96-1.11 ± 0.92
51-3.153.641.06
650.04 ± 2.893.32 ± 1.41-0.28 ± 0.77
79-1.51 ± 1.262.34 ± 2.580.11 ± 0.90
860.06 ± 1.734.47 ± 2.241.22 ± 1.15
950.82 ± 3.615.29 ± 3.080.90 ± 1.07
1061.38 ± 1.084.93 ± 1.080.43 ± 0.51

On the two task groups with fewer forecasts, the gap grows by 4 to 5 points from K=4K = 4 to K=10K = 10, for both metrics. On the tasks with the most forecasts, its absolute value never exceeds 1.22 points. Random portfolios thus show the gap, and the gap follows the number of ensemble members and the size of the task, not the search. The cause is the in-sample fit of the ensemble weights: with more members, there are more weights to fit, and with fewer forecasts, the fit is less reliable (Caruana et al., 2006; Cawley & Talbot, 2010).

This optimism is therefore a problem of ensembles for time series forecasting in general, and not a problem of Auto-Chroma alone. Every method in our comparison (Chroma, the heuristic portfolios, and all search strategies) fits its weights in the same way. Recent work combines TSFMs in different ways: Synapse (Das et al., 2026) weights its members by their errors on a recent window, TimeCopilot (Garza & Rosillo, 2025) uses a median ensemble of three models, and MoiraiAgent (Yan et al., 2026) trains a selector that picks one expert for each forecasting task. These works report test results, and their descriptions state no validation split for their design choices, so how well a selection signal carries over to test stays unexamined. Our paper fixes the validation split, optimizes on it alone, and evaluates every candidate portfolio on both splits, which shows when the validation signal carries over and when it does not (RQ5). Given the rise of TSFM ensembles, we believe that this analysis is necessary.

This agrees with the analysis of Section 4 (Figure 7 and its appendix): the negative validation–test correlation is on the tasks with few forecasts of both benchmarks, and it fades as the number of forecasts grows. We therefore conclude that the gap comes from the lack of validation signal on small tasks, which any model selection on this holdout inherits, rather than from the search.

W3: Contribution of the components

Ablations in the main paper. We followed the suggestion of the reviewer and moved these results from the appendix to the main text of the revised paper: the analyst ablation is now RQ6 (Section 4.6, Table 1), and the cross-benchmark transfer is now RQ7 (Section 4.7, Table 2).

Analyst. We agree that removing the analyst gives comparable accuracy, and the paper explicitly says so in RQ6. We retain the analyst because its reports make each round interpretable: every report verifies the hypotheses of the preceding strategist report against the measured outcome and records the result.

Memory and reasoning. We ran two new experiments on fev-bench-mini that remove parts of the memory of the strategist, and we compare them with the search of the paper.

AuthorsOpenReview ↗

Continue from the 2nd comment

(i) Without memory, the strategist is unreliable. In the zero-shot setting, the strategist has no search history: it sees only the two heuristic portfolios and their results, and it makes one proposal. We repeated this with 10 seeds for each model.

Validation skill score, SQL (%), against the frequency heuristic:

ModelZero-shot: minZero-shot: maxZero-shot: meanZero-shot: stdAuto-Chroma
Chroma-4M1.963.942.660.695.41
Chronos-2-Small0.403.001.760.733.67

The proposals vary widely between seeds: for Chronos-2-Small, the standard deviation is about 40% of the mean. Even the best of the 10 proposals stays below Auto-Chroma for both models, and the mean is about half of it. The prior knowledge of the LLM alone is therefore not sufficient, and the memory is necessary.

(ii) With the raw history alone, a minimal strategist is already effective. In the second setting, the strategist runs the full 40 rounds with minimal instructions. It knows only the composition and the performance of the past portfolios: it writes no reports, there is no analyst, and it has no access to the ensemble weights. The LLM, the budget, and the benchmark are the same as in the paper. The table reports the best validation skill score (SQL, %, against the frequency heuristic) of each search.

ModelMinimal strategistAuto-Chroma
Chroma-4M3.545.41
Chronos-2-Small4.203.67

For Chronos-2-Small, the minimal strategist is on par with the Auto-Chroma loop. For Chroma-4M, it ends lower than the Auto-Chroma loop. The reasoning of the LLM over the history can therefore be effective even without reports, detailed instructions, or ensemble weights.

Together with the comparison with greedy and evolutionary search (W4 below), these results separate the two components. Greedy and evolutionary search also retain past configurations and numerical results, but they use them through fixed rules, and they end below Auto-Chroma. The zero-shot strategist reasons without memory, and it is unreliable. The memory is therefore necessary, and the reasoning of the LLM over it is effective even in its minimal form. This is plausible: the reports are an explicit form of the reasoning of the LLM, and without them the strategist can still analyze the past portfolios and their results by itself in every round. The reports serve two other purposes. First, they make the search interpretable: each report records why a portfolio was proposed and whether its hypothesis held. Second, they save tokens: the patterns found in earlier rounds persist in the reports, so the strategist does not deduce them again from the raw results in each round. This agrees with RQ6, where removing the analyst gives comparable accuracy. We will state this role of the reports more clearly in the final camera-ready version.

Cross-benchmark transfer. This experiment (RQ7, Table 2, reproduced in W2) tests whether a portfolio that Auto-Chroma finds is specific to the benchmark on which the search ran, or general. Both rows are portfolios that Auto-Chroma found: the first by a search on the target benchmark, the second by a search on the other benchmark, with only the ensemble weights refit. The two portfolios perform comparably, so we do not conclude that one is better than the other. Our conclusion is that the portfolios found by the search are general: one search can serve more than one benchmark.

W4: Stronger search baselines

Evolutionary search. We added evolutionary search as a new baseline in the revised paper. It is a direct generalization of our greedy baseline. We use a steady-state (μ+1)(\mu+1) evolutionary algorithm: a population of at most μ=5\mu = 5 portfolios starts from the two heuristic portfolios; in every round, parents are picked by tournaments of size 2, two parents are recombined by a uniform crossover over their specialists with probability 0.50.5, and the child receives one mutation of greedy search. Everything else is matched with Auto-Chroma, greedy, and random search: the budget of 40 candidates, the post-training and evaluation code, the objective, and the acceptance rule. The pseudocode is the new Algorithm 7 in the appendix, and the new curves are in the updated Figure 4 (Chroma-4M) and Figure 10 (Chronos-2-Small).

The table reports, on fev-bench-mini, the best validation skill score (SQL, against the frequency heuristic) after 40 candidates and the number of unique specialists that each method post-trains. As in the right panel of Figure 4, the count includes the 11 specialists of the two heuristic portfolios.

Reviewer_TFq8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Auto-Chroma automates the construction of portfolios of small time-series specialists. An LLM analyst and strategist use a persistent history of configurations, validation results, routing weights, and reports to propose each specialist’s training-data subset, learning rate, and training steps. A fixed pipeline post-trains and caches specialists, fits task-specific greedy ensemble weights on validation windows, and evaluates candidate portfolios over 40 search rounds. LLM agents operate during offline search, while only selected forecasting models run at inference. Across three small generalists and two benchmarks, the selected portfolios improve the primary test metric over frequency-based portfolios in five of six settings, generally modestly. Additional analyses examine search efficiency, validation–test mismatch, portfolio transfer, and the mixed effect of a separate analyst.

Critical strengths
  1. The problem setting is practically motivated, and the framework provides a complete workflow for replacing fixed frequency- or domain-based specialist groupings with a search over data subsets and training configurations. Cached specialists can be reused, and LLMs operate only during offline design rather than forecasting. Improvements over the frequency heuristic in five of six settings provide initial evidence of effectiveness, although most gains are modest.

  2. The evaluation transparently examines both strengths and limitations through comparisons across orchestrator LLMs, repeated searches, cross-benchmark transfer, memory-use analysis, and analyst ablations. Reporting the validation–test mismatch and the comparable performance without a separate analyst helps readers assess which aspects are supported and which remain uncertain.

Critical weaknesses
  1. The evaluation lacks comparisons with Bayesian optimization or evolutionary search under matched computational budgets. Consequently, the necessity of LLM-guided search and the additional contributions of its reasoning and memory mechanisms remain insufficiently established.

  2. Repeatedly using the same validation data both to fit ensemble weights and to search over portfolios risks overfitting the validation set. Although the authors analyze the validation–test discrepancy, they do not validate an effective mitigation, leaving it unclear whether the search gains generalize reliably to unseen data.

Questions for authors
  1. How does Auto-Chroma compare with Bayesian optimization or evolutionary search under matched search spaces and total computational budgets? What evidence isolates the additional value of LLM reasoning and memory beyond retaining past configurations and numerical evaluation results?

  2. Given that the same validation data are used to fit ensemble weights and guide portfolio search, what evidence distinguishes validation overfitting from other causes of the validation–test discrepancy? Have you evaluated separate temporal splits for weight fitting and portfolio selection, or nested temporal validation, to test whether the search gains generalize reliably?

5 comments

We thank you for your constructive feedback. We hope that our responses address your concerns and merit a higher score. All changes in the revised paper are marked in blue.

W1/Q1: Comparison with Bayesian optimization and evolutionary search

Bayesian optimization (BO). BO repeats two steps. First, it fits a predictive model of the score to the candidates evaluated so far. Second, it selects the next candidate as the one that this model rates as most promising. This works well for hyperparameter tuning, because the input is a fixed list of named hyperparameters, the search space is small, and the most promising candidate can be found by sampling. Portfolio search has none of these properties. A portfolio is a set of specialists of variable size, each with its own dataset subset and training hyperparameters, and BO over such spaces is still an open problem (COMBO, Oh et al., 2019; set kernels, Buathong et al., 2020). A BO method for our task would have to solve three problems:

  • (i) Compare two portfolios. The predictive model needs a measure of similarity between candidates. A portfolio has between 1 and 10 specialists, their order has no meaning, and each specialist combines a discrete choice (a subset of the datasets) with continuous ones (learning rate and step count). No standard similarity measure covers this input.
  • (ii) Predict a score that the specialists produce together. The score of a portfolio is not the sum of the scores of its specialists. Ensemble selection fits the weights per task, so a specialist that receives a large weight in one portfolio can receive zero weight in another portfolio that already covers the same tasks. BO observes only one score S(P)S(P) per portfolio. To predict an unseen portfolio, the model must therefore infer from these scores how much each specialist contributed, and in combination with which other specialists.
  • (iii) Find the next candidate. The space is too large for sampling. The corpus has 44 datasets, so one specialist chooses among 244−1≈1.8×10132^{44}-1 \approx 1.8 \times 10^{13} non-empty dataset subsets. A portfolio of 10 specialists chooses 10 distinct subsets, which gives (244−110)≈10126\binom{2^{44}-1}{10} \approx 10^{126} portfolios, before the learning rate and the step count of each specialist are chosen. The space is also discrete, so the most promising candidate cannot be found with gradients or by enumeration.

Set kernels address (i), but (ii) and (iii) remain non-trivial. We therefore see no off-the-shelf BO baseline for this problem; building one would be a research contribution of its own.

Evolutionary search. Evolutionary search is a direct generalization of our greedy baseline, so we added it as a steady-state (μ+1)(\mu+1) evolutionary algorithm: a population of at most μ=5\mu = 5 portfolios starts from the two heuristic portfolios; in every round, parents are picked by tournaments of size 2, two parents are recombined by a uniform crossover over their specialists with probability 0.50.5, and the child receives one mutation of greedy search. Everything else is matched with Auto-Chroma, greedy, and random search: the budget of 40 candidates, the post-training and evaluation code, the objective, and the acceptance rule. The pseudocode is the new Algorithm 7 in the appendix, and the new curves are in the updated Figure 4 (Chroma-4M) and Figure 10 (Chronos-2-Small).

The table reports, on fev-bench-mini, the best validation skill score (SQL, against the frequency heuristic) after 40 candidates and the number of unique specialists that each method post-trains. As in the right panel of Figure 4, the count includes the 11 specialists of the two heuristic portfolios.

ModelMethodBest validation skill score, SQL (%)Unique specialists
Chroma-4MAuto-Chroma5.4157
Chroma-4MGreedy3.0549
Chroma-4MEvolutionary2.4053
Chroma-4MRandom3.57290
Chronos-2-SmallAuto-Chroma3.6755
Chronos-2-SmallGreedy3.5351
Chronos-2-SmallEvolutionary2.2254
Chronos-2-SmallRandom2.64290

Evolutionary search ends below Auto-Chroma for both models, at a comparable number of post-trained specialists, so the comparison is also matched in training compute. It also ends below greedy and random search: every candidate mutates or recombines members of the current population, and the poor portfolios of the initial population leave only one at a time, so its running best rises slowly (Section 4, RQ2).

AuthorsOpenReview ↗

Continue from the 1st Comment

W1/Q1 (continued): Value of LLM reasoning and memory

We ran two new experiments on fev-bench-mini that remove parts of the memory of the strategist, and we compare them with the search of the paper.

(i) Without memory, the strategist is unreliable. In the zero-shot setting, the strategist has no search history: it sees only the two heuristic portfolios and their results, and it makes one proposal. We repeated this with 10 seeds for each model.

Validation skill score, SQL (%), against the frequency heuristic:

ModelZero-shot: minZero-shot: maxZero-shot: meanZero-shot: stdAuto-Chroma
Chroma-4M1.963.942.660.695.41
Chronos-2-Small0.403.001.760.733.67

The proposals vary widely between seeds: for Chronos-2-Small, the standard deviation is about 40% of the mean. Even the best of the 10 proposals stays below Auto-Chroma for both models, and the mean is about half of it. The prior knowledge of the LLM alone is therefore not sufficient, and the memory is necessary.

(ii) With the raw history alone, a minimal strategist is already effective. In the second setting, the strategist runs the full 40 rounds with minimal instructions. It knows only the composition and the performance of the past portfolios: it writes no reports, there is no analyst, and it has no access to the ensemble weights. The LLM, the budget, and the benchmark are the same as in the paper. The table reports the best validation skill score (SQL, %, against the frequency heuristic) of each search.

ModelMinimal strategistAuto-Chroma
Chroma-4M3.545.41
Chronos-2-Small4.203.67

For Chronos-2-Small, the minimal strategist is on par with the Auto-Chroma loop. For Chroma-4M, it ends lower than the Auto-Chroma loop. The reasoning of the LLM over the history can therefore be effective even without reports, detailed instructions, or ensemble weights.

Together with the comparison above, these results separate the two components. Greedy and evolutionary search also retain past configurations and numerical results, but they use them through fixed rules, and they end below Auto-Chroma. The zero-shot strategist reasons without memory, and it is unreliable. The memory is therefore necessary, and the reasoning of the LLM over it is effective even in its minimal form. This is plausible: the reports are an explicit form of the reasoning of the LLM, and without them the strategist can still analyze the past portfolios and their results by itself in every round. The reports serve two other purposes. First, they make the search interpretable: each report records why a portfolio was proposed and whether its hypothesis held. Second, they save tokens: the patterns found in earlier rounds persist in the reports, so the strategist does not deduce them again from the raw results in each round. This agrees with RQ6, where removing the analyst gives comparable accuracy. We will state this role of the reports more clearly in the final camera-ready version.

W2/Q2: Validation overfitting and nested temporal validation

We did not evaluate a separate temporal split or nested validation. We explain both choices before the evidence.

Why no third split. A third temporal split holds out one more block of every series. It shortens the context of every task, removes more of the short series, and departs from the protocol of Chroma, AutoGluon, and the benchmarks, so the scores would no longer be comparable with the heuristic portfolios and the published models. The shared holdout is thus a property of the protocol that all compared methods use (Auto-Chroma, greedy, random, evolutionary, and the heuristics), not of Auto-Chroma.

Why no nested validation. We fit the ensemble weights as AutoGluon does by default, on one validation window; Appendix B gives the details. Nested validation needs more than one validation window for each test window, and no standard protocol defines how to do this on a benchmark with rolling test windows. We therefore kept the default. We agree that nested validation is a promising direction, and we will name it as future work in the limitations of the final camera-ready version.

What separates search overfitting from other causes of the gap. As in RQ5 of the paper, our position is that the gap comes from a weak validation signal and not from the search: tasks that score few forecasts give noisy validation scores, so validation and test performance correlate poorly on them. Two results support this position.

(i) The portfolios transfer across benchmarks (RQ7, Table 2, reproduced here). We evaluate the best portfolio of each search on the benchmark that it was not searched on, with only the ensemble weights refit. The search never scored this portfolio on the target benchmark, so a portfolio that was overfit to its own validation set would lose here.

AuthorsOpenReview ↗

Continue from the 2nd comment

Test skill score against Seasonal Naive (%), higher is better:

ModelPortfolioCB-II MASECB-II WQLfev-bench-mini SQLfev-bench-mini MASE
Chroma-4MSearched on target19.0935.3936.1623.16
Chroma-4MTransferred from other18.5536.0436.6223.66
Chronos-2-SmallSearched on target25.3242.1949.2438.63
Chronos-2-SmallTransferred from other25.1642.0949.3938.78

The transferred portfolio is on par with the portfolio searched on the target. The portfolios found by the search are therefore not overfit to the validation set, since they generalize to a new benchmark without loss.

(ii) The gap appears without any search. To separate the effect of the search from the effect of the ensemble, we need portfolios that no search selected. Random search provides them: it draws the number of specialists KK and their configurations without looking at any score. We take its 40 portfolios for Chronos-2-Small on fev-bench-mini and compute, for each one, the gap between its validation and its test skill score (validation minus test, in points). A positive gap means that the portfolio looks better on validation than on test. To observe the effect of the number of forecasts on the validation signal, we divide the 20 tasks into the same three groups as the paper uses for this benchmark (Appendix D.5), from the fewest scored forecasts to the most, and report the gap for each KK. If the search caused the gap, these portfolios would show none.

Gap in points, SQL (mean ± standard deviation over nn portfolios):

KKnn8 tasks, 20–160 forecasts6 tasks, 240–420 forecasts6 tasks, 700–11150 forecasts
48-2.79 ± 1.61-0.18 ± 1.67-1.22 ± 0.48
51-2.053.331.21
650.38 ± 2.623.15 ± 1.31-0.62 ± 1.01
79-0.92 ± 1.222.48 ± 2.31-0.27 ± 0.54
860.66 ± 1.754.40 ± 1.950.70 ± 1.22
951.01 ± 3.085.14 ± 3.130.36 ± 1.14
1061.45 ± 0.974.81 ± 0.83-0.17 ± 0.47

Gap in points, MASE:

KKnn8 tasks, 20–160 forecasts6 tasks, 240–420 forecasts6 tasks, 700–11150 forecasts
48-3.28 ± 1.86-0.12 ± 1.96-1.11 ± 0.92
51-3.153.641.06
650.04 ± 2.893.32 ± 1.41-0.28 ± 0.77
79-1.51 ± 1.262.34 ± 2.580.11 ± 0.90
860.06 ± 1.734.47 ± 2.241.22 ± 1.15
950.82 ± 3.615.29 ± 3.080.90 ± 1.07
1061.38 ± 1.084.93 ± 1.080.43 ± 0.51

On the two task groups with fewer forecasts, the gap grows by 4 to 5 points from K=4K = 4 to K=10K = 10, for both metrics. On the tasks with the most forecasts, its absolute value never exceeds 1.22 points. Random portfolios thus show the gap, and the gap follows the number of ensemble members and the size of the task, not the search. The cause is the in-sample fit of the ensemble weights: with more members, there are more weights to fit, and with fewer forecasts, the fit is less reliable (Caruana et al., 2006; Cawley & Talbot, 2010).

This optimism is therefore a problem of ensembles for time series forecasting in general, and not a problem of Auto-Chroma alone. Every method in our comparison (Chroma, the heuristic portfolios, and all search strategies) fits its weights in the same way. Recent work combines TSFMs in different ways: Synapse (Das et al., 2026) weights its members by their errors on a recent window, TimeCopilot (Garza & Rosillo, 2025) uses a median ensemble of three models, and MoiraiAgent (Yan et al., 2026) trains a selector that picks one expert for each forecasting task. These works report test results, and their descriptions state no validation split for their design choices, so how well a selection signal carries over to test stays unexamined. Our paper fixes the validation split, optimizes on it alone, and evaluates every candidate portfolio on both splits, which shows when the validation signal carries over and when it does not (RQ5). Given the rise of TSFM ensembles, we believe that this analysis is necessary.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary

Dear AC and Reviewers,

Thank you for your time and valuable feedback. We appreciate that Reviewer TFq8 found the problem setting practically motivated, and that Reviewer cDGq found the formulation of portfolio construction as a search well motivated and its insights practical. We are especially grateful to Reviewer sAr5, whose two rounds of follow-up helped us sharpen the paper. The reviewer found that our additional results answered the follow-up requests and regarded the openness of the study as its main strength. The revised PDF states the requested points explicitly. We believe that we have addressed all concerns of the three reviewers, including those on which we have not yet received a follow-up.

We would like to briefly restate what this paper contributes.

Portfolios of small specialists can match time series foundation models that are orders of magnitude larger, but their design is still manual, with the data partitioned once by frequency or domain. Auto-Chroma casts portfolio design as a search and runs it end to end with memory-augmented LLM agents, which propose each portfolio from the full history of the search. To our knowledge, this is the first work to treat portfolio construction as an autonomous search and to study the search itself.

Our results show that the search is effective and efficient. On the validation objective, Auto-Chroma leads greedy, evolutionary, and random search under the same budget, and it surpasses the final performance of random search within a quarter of the budget with about 5x fewer post-trained specialists. On test, the portfolios outperform the frequency heuristic in five of the six settings with mild gains and remain competitive with much larger models. Beyond performance, we show that heuristic partitions can degrade when data is added, that the portfolios transfer across benchmarks, and that the standard holdout gives an unreliable selection signal on small tasks, a pitfall for any holdout-based model selection.

During the rebuttal, we addressed the following concerns (all changes are marked in blue):

  1. Stronger search baselines (TFq8, cDGq, sAr5). Evolutionary search under the same budget, objective, and protocol (Section 4.2, Appendix D.2, Algorithm 7).
  2. Value of memory and LLM reasoning (TFq8, cDGq, sAr5). Ablations without memory and with a minimal strategist (Appendix E.2, Table 11).
  3. Role of the analyst and the reports (cDGq, sAr5). Ablations moved to the main text as RQ6 and RQ7; the reports add interpretability rather than accuracy (Section 4.6, Appendix E.2).
  4. Robustness of the test gains (cDGq, sAr5). Bootstrap confidence intervals (Appendix D.4, Table 6), three repeated runs (Appendix E.1, Table 10), and test gains stated as mild in the abstract and RQ4.
  5. Validation overfitting (TFq8, cDGq, sAr5). Cross-benchmark transfer (Table 2), the validation-test gap of random portfolios (Table 8), and, for the follow-up of sAr5, the test results and gaps of all search strategies (Appendix D.6, Tables 7 and 9, Figure 15).
  6. Validation protocol (TFq8, sAr5). A separate selection split and nested validation named as future work (Section 5).
  7. Search cost (sAr5). FLOPs and wall-clock time per round (Appendix E.3, Tables 12 and 13).
  8. Other LLM-based search methods (cDGq). To our knowledge, none exists for portfolios of specialists (Section 2).

We believe this work is a first step toward the fully automated design of model portfolios, and that its findings on holdout-based selection and on agentic search apply well beyond Chroma. We recognize that the test gains are mild and that more reliable validation signals are an important next step, and the paper reports both openly.

Thank you again for your careful consideration. We hope the contributions and the additional evidence from the rebuttal will be taken into account in the final decision.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.