- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Auto-Chroma, an agentic framework for automating the construction of portfolios of small time-series forecasting specialists. Starting from a compact pretrained generalist, Auto-Chroma searches over specialist configurations, including the post-training datasets, learning rate, and training duration. An analyst LLM reviews the latest results and prior search history, while a strategist LLM proposes subsequent portfolio configurations. A persistent database stores configurations, scores, routing weights, and agent reports. Numerical specialist training, ensemble calibration, and portfolio evaluation remain outside the LLM components.
The method is evaluated on CB-II and fev-bench-mini using three relatively small forecasting generalists, including both univariate and multivariate models. Auto-Chroma is compared with frequency-based and domain-based specialist portfolios, as well as random and greedy search. The proposed method outperforms the frequency-based portfolio on the reported test metric in five of six model–benchmark settings. The paper additionally examines validation-to-test transfer, portfolio similarity across independent runs and orchestrator LLMs, cross-benchmark transfer, use of prior search history, the contribution of the analyst, and computational cost.
Overall, the paper presents a useful extension of specialist-based forecasting ensembles and provides an unusually broad and candid analysis of the search process. However, the experiments establish more clearly that the overall system can find strong portfolios than that its particular memory-augmented, two-agent architecture is responsible for the improvement.
Critical strengths
1. Useful extension of fixed specialist portfolios
The central idea is well motivated. Rather than treating experts as fixed pretrained forecasters and learning only how to route among them, Auto-Chroma searches over what the specialists learn. The method selects each specialist’s post-training datasets and training hyperparameters, thereby changing the individual experts rather than merely changing their ensemble weights. This is a meaningful extension of common forecasting-ensemble approaches based on immutable pretrained experts. The search problem is also practically relevant because smaller specialist portfolios may provide a better accuracy–efficiency tradeoff than substantially larger general-purpose forecasting models.
2. Encouraging improvements over the main heuristic baseline
The most informative comparisons are against the frequency-based specialist portfolio rather than against the initial generalist alone. On CB-II, test MASE skill increases from 17.56 to 19.09 for Chroma-4M and from 19.35 to 19.98 for Chronos-Bolt-Tiny. These results indicate that searching over specialist construction can outperform a reasonable manually designed portfolio. The evaluation also includes three small generalists and covers both univariate and multivariate settings. Auto-Chroma improves over the frequency-based portfolio in five of the six reported model–benchmark combinations. Although some improvements are small, the overall result provides credible evidence that automated specialist construction is useful.
3. Unusually candid analysis of negative results
A major strength is that the paper reports analyses that do not consistently favor the proposed method. The validation–test analysis identifies a failure of the search objective for Chronos-2-Small on fev-bench-mini, rather than presenting only the successful settings. The analyst ablation similarly shows that removing the analyst performs comparably and sometimes better. The cross-benchmark transfer experiment shows that fixed specialist portfolios transferred between benchmarks, with only the ensemble refitted, produce very similar results. These findings weaken some of the potential architectural claims, but their inclusion substantially improves the scientific quality and credibility of the paper. The paper is more informative because it distinguishes positive headline results from limitations observed during analysis.
4. The search process is treated as an object of study
The paper goes beyond reporting only the best portfolio. It analyzes similarities between portfolios found across runs and orchestrator LLMs, dataset coverage, specialist reuse, references to earlier candidates, validation-to-test behavior, and transfer across benchmarks. This provides a richer picture of how the search behaves and makes the system easier to understand and audit. The persistent record of configurations, numerical outcomes, routing weights, and reports may itself be useful operationally, even if its incremental contribution to forecast accuracy is not yet established.
5. Appropriate separation between LLM reasoning and numerical computation
The LLM components propose and analyze portfolio configurations, but specialist training, routing calibration, and evaluation remain outside the LLM. This is a sensible system design because the language models are used for hypothesis generation and search orchestration rather than numerical forecasting or metric calculation.
Critical weaknesses
1. The robustness of the test improvements is not fully established
Several improvements over the frequency-based portfolio are small. Chronos-2-Small improves by only 0.08 MASE skill points on CB-II. On fev-bench-mini, Chroma-4M and Chronos-Bolt-Tiny gain only 0.28 and 0.16 SQL skill points, respectively, while Chronos-2-Small loses 0.51 SQL skill points.
The larger CB-II improvements for the two smaller generalists are encouraging. However, the aggregate observation that Auto-Chroma wins in five of six settings does not establish that the smaller gains would reliably repeat across independent searches, alternative benchmark samples, or different random seeds.
The appendix reports similarity between independently discovered portfolios, which is useful information about search behavior. However, similarity in portfolio composition does not directly quantify variation in final test performance. Repeated end-to-end searches reporting the distribution of test improvements would provide stronger support for the robustness of the results.
2. The contribution of report-based memory is not demonstrated
Figure 16 and Appendix E.2 count references to earlier candidates in the agent reports. This establishes that the agents use the available history, but it does not show that using this history improves portfolio quality. A more direct experiment would compare the full report-based memory with reduced-memory alternatives under the same search budget. Useful comparisons could include:
- Configurations and numerical scores without natural-language reports
- Only the most recent search round
- A fixed-size history
- Summarized rather than complete history
- No search history
Without such an ablation, it remains unclear whether persistent report-based memory produces better portfolios or mainly provides a more interpretable record of the search.
3. The separate analyst does not show a consistent accuracy benefit
The analyst ablation raises questions about the proposed two-agent architecture. Removing the analyst increases Chroma-4M CB-II test WQL skill from 35.39 to 37.40, and neither architecture consistently dominates in the remaining settings.
The analyst may still contribute useful diagnostics, auditability, or structured reporting. However, the current experiments do not demonstrate that the analyst improves forecasting accuracy or search efficiency. The paper should either provide stronger evidence for the analyst or narrow its claim, positioning the analyst principally as an interpretability and search-documentation component.
4. The LLM-free baselines are not fully capability-matched
The random and greedy baselines are informative, but they do not cleanly isolate the value of LLM guidance.
Random search restricts specialist subsets to at most six datasets, while Auto-Chroma permits arbitrary nonempty subsets. Auto-Chroma can also use cached specialists, previous results, and per-task routing diagnostics to recombine or modify earlier configurations. The compared methods therefore differ in search space, information access, specialist reuse, and proposal mechanism, rather than differing only in whether an LLM generates the candidate.
A stronger comparison would give a conventional optimizer the same configuration space, specialist cache, numerical feedback, proposal budget, and portfolio-evaluation mechanism. Bayesian optimization, evolutionary search, bandit-based search, or a structured mutation strategy would help determine whether the advantage comes specifically from LLM guidance.
Greedy search also approaches Auto-Chroma late in the Chronos-2-Small budget. This suggests that part of the advantage may arise from broader proposal operators or faster exploration rather than from the full agent architecture.
5. Search costs are not comprehensively matched
The paper reports portfolio-evaluation budgets, numbers of unique specialists, and an LLM API cost of USD 1.96. These are useful measurements, but they do not fully represent the cost of the search.
Different methods may train different numbers of specialists, reuse cached specialists at different rates, or incur different routing-calibration and evaluation costs. Equal numbers of portfolio evaluations therefore do not necessarily correspond to equal total computation. The reported LLM cost also excludes specialist training and routing calibration.
Independent runs matched by accelerator time, wall-clock time, FLOPs, or total measured compute would provide a stronger basis for the efficiency claim.
Questions for authors
-
Repeated-search uncertainty: What is the distribution of final test performance across independent Auto-Chroma runs? In particular, how often do the smaller improvements over the frequency-based portfolio change sign, and are the reported gains statistically distinguishable from run-to-run variation?
-
Value of report-based memory:Can the authors compare the full report-based memory against simpler alternatives under the same search budget, such as configurations and numerical outcomes only, recent-round-only context, a fixed-size history, or no history? The current analysis shows that agents reference previous candidates, but not whether this improves portfolio quality.
-
Role of the analyst: Given that removing the analyst is competitive and improves some reported results, what claim do the authors intend to make about this component? Is its primary contribution forecasting accuracy, search efficiency, interpretability, or auditability?
-
Capability- and cost-matched non-LLM search: Can the authors compare Auto-Chroma with a conventional optimizer that has the same search space, specialist cache, numerical and per-task feedback, proposal budget, and total computational budget? Please include specialist-training and routing-calibration costs, since equal portfolio evaluations may not imply equal overall compute.
10 comments
Continue from the first comment
(ii) With the raw history alone, a minimal strategist is already effective. In the second setting, the strategist runs the full 40 rounds with minimal instructions. It knows only the composition and the performance of the past portfolios: it writes no reports, there is no analyst, and it has no access to the ensemble weights. The LLM, the budget, and the benchmark are the same as in the paper. The table reports the best validation skill score (SQL, %, against the frequency heuristic) of each search.
| Model | Minimal strategist | Auto-Chroma |
|---|---|---|
| Chroma-4M | 3.54 | 5.41 |
| Chronos-2-Small | 4.20 | 3.67 |
For Chronos-2-Small, the minimal strategist is on par with the Auto-Chroma loop. For Chroma-4M, it ends lower than the Auto-Chroma loop. The reasoning of the LLM over the history can therefore be effective even without reports, detailed instructions, or ensemble weights.
Together with the comparison with greedy and evolutionary search (W4 below), these results separate the two components. Greedy and evolutionary search also retain past configurations and numerical results, but they use them through fixed rules, and they end below Auto-Chroma. The zero-shot strategist reasons without memory, and it is unreliable. The memory is therefore necessary, and the reasoning of the LLM over it is effective even in its minimal form. This is plausible: the reports are an explicit form of the reasoning of the LLM, and without them the strategist can still analyze the past portfolios and their results by itself in every round. The reports serve two other purposes. First, they make the search interpretable: each report records why a portfolio was proposed and whether its hypothesis held. Second, they save tokens: the patterns found in earlier rounds persist in the reports, so the strategist does not deduce them again from the raw results in each round. This agrees with RQ6, where removing the analyst gives comparable accuracy. We will state this role of the reports more clearly in the final camera-ready version.
W3/Q3: Role of the analyst
We agree with the reviewer. The paper already states this role of the analyst in RQ6 (Section 4.6, Table 1), which we moved from the appendix to the main text of the revised paper. RQ6 reports that removing the analyst gives comparable accuracy, and that we retain the analyst because its reports make each round interpretable: every report verifies the hypotheses of the preceding strategist report against the measured outcome and records the result. The primary contribution of the analyst is therefore interpretability and auditability, and we do not claim that it improves forecasting accuracy. Its reports also have the practical role that we describe in W2: they persist in the memory, so the agents in later rounds read the verified outcome of each earlier design and do not spend tokens to deduce it again from the raw results. We will make this point clearer in the final camera-ready version.
W4/Q4: Search space and information of the LLM-free baselines
The LLM-free strategies differ from Auto-Chroma only in the proposal step: the budget of 40 candidates, the post-training and evaluation code, the cache of trained specialists, the objective, and the acceptance rule are identical.
Search space of random search. Random search draws each specialist from at most six datasets on purpose. The alternative is a uniform draw over all non-empty subsets of the 44 datasets. Such a draw includes each dataset with probability , so the size of the subset concentrates around 22 datasets, and a subset of at most six datasets has a probability of about . This search would post-train only specialists on about half of the corpus, which are close to the generalist, and it would not test which division of the corpus is useful. The limit of six datasets is therefore the prior that makes random search a meaningful baseline, and not a restriction that favors Auto-Chroma. Random search also post-trains far more unique specialists (290) than Auto-Chroma, greedy search, and evolutionary search (49 to 57), so it is the most costly of the four strategies (see the updated Figure 4 and Figure 10).
Greedy and evolutionary search are not restricted. Both start from the heuristic portfolios, whose specialists have no such limit (the frequency heuristic divides the 44 datasets among 5 specialists, the largest with 16 datasets), and their operators add, move, and merge datasets without a limit on the size of a subset (Appendix D.2). They can therefore reach any non-empty subset, as Auto-Chroma can.
Continue from the second comment
W4/Q4 (continued): Comparison with Bayesian optimization and evolutionary search
Bayesian optimization (BO). BO repeats two steps. First, it fits a predictive model of the score to the candidates evaluated so far. Second, it selects the next candidate as the one that this model rates as most promising. This works well for hyperparameter tuning, because the input is a fixed list of named hyperparameters, the search space is small, and the most promising candidate can be found by sampling. Portfolio search has none of these properties. A portfolio is a set of specialists of variable size, each with its own dataset subset and training hyperparameters, and BO over such spaces is still an open problem (COMBO, Oh et al., 2019; set kernels, Buathong et al., 2020). A BO method for our task would have to solve three problems:
- (i) Compare two portfolios. The predictive model needs a measure of similarity between candidates. A portfolio has between 1 and 10 specialists, their order has no meaning, and each specialist combines a discrete choice (a subset of the datasets) with continuous ones (learning rate and step count). No standard similarity measure covers this input.
- (ii) Predict a score that the specialists produce together. The score of a portfolio is not the sum of the scores of its specialists. Ensemble selection fits the weights per task, so a specialist that receives a large weight in one portfolio can receive zero weight in another portfolio that already covers the same tasks. BO observes only one score per portfolio. To predict an unseen portfolio, the model must therefore infer from these scores how much each specialist contributed, and in combination with which other specialists.
- (iii) Find the next candidate. The space is too large for sampling. The corpus has 44 datasets, so one specialist chooses among non-empty dataset subsets. A portfolio of 10 specialists chooses 10 distinct subsets, which gives portfolios, before the learning rate and the step count of each specialist are chosen. The space is also discrete, so the most promising candidate cannot be found with gradients or by enumeration.
Set kernels address (i), but (ii) and (iii) remain non-trivial. We therefore see no off-the-shelf BO baseline for this problem; building one would be a research contribution of its own.
Evolutionary search. Evolutionary search is a direct generalization of our greedy baseline, so we added it as a steady-state evolutionary algorithm: a population of at most portfolios starts from the two heuristic portfolios; in every round, parents are picked by tournaments of size 2, two parents are recombined by a uniform crossover over their specialists with probability , and the child receives one mutation of greedy search. Everything else is matched with Auto-Chroma, greedy, and random search: the budget of 40 candidates, the post-training and evaluation code, the objective, and the acceptance rule. The pseudocode is the new Algorithm 7 in the appendix, and the new curves are in the updated Figure 4 (Chroma-4M) and Figure 10 (Chronos-2-Small).
The table reports, on fev-bench-mini, the best validation skill score (SQL, against the frequency heuristic) after 40 candidates and the number of unique specialists that each method post-trains. As in the right panel of Figure 4, the count includes the 11 specialists of the two heuristic portfolios.
| Model | Method | Best validation skill score, SQL (%) | Unique specialists |
|---|---|---|---|
| Chroma-4M | Auto-Chroma | 5.41 | 57 |
| Chroma-4M | Greedy | 3.05 | 49 |
| Chroma-4M | Evolutionary | 2.40 | 53 |
| Chroma-4M | Random | 3.57 | 290 |
| Chronos-2-Small | Auto-Chroma | 3.67 | 55 |
| Chronos-2-Small | Greedy | 3.53 | 51 |
| Chronos-2-Small | Evolutionary | 2.22 | 54 |
| Chronos-2-Small | Random | 2.64 | 290 |
Evolutionary search ends below Auto-Chroma for both models, at a comparable number of post-trained specialists, so the comparison is also matched in training compute. It also ends below greedy and random search: every candidate mutates or recombines members of the current population, and the poor portfolios of the initial population leave only one at a time, so its running best rises slowly (Section 4, RQ2).
W5/Q4: Search cost
Training. The cost of a portfolio depends on the number of its specialists, and a cached specialist is not trained again. The number of unique specialists is therefore the measure of the training compute of a search, and the table in W4 gives it for every method. Auto-Chroma, greedy search, and evolutionary search have comparable training budgets, while the training budget of random search is significantly higher.
We appreciate your careful assessment of our work. We hope that the new experiments and clarifications strengthen your confidence in the paper, and we kindly ask you to consider a higher score. All changes in the revised paper are marked in blue.
W1/Q1: Robustness of the test improvements
Repeated searches. Appendix E.1 compares the portfolios of three independent searches with an identical configuration on
fev-bench-mini. We have now evaluated all three on the test split (test skill score against Seasonal Naive, SQL, %; search 1 is the one that the paper reports):Auto-Chroma is above the frequency heuristic in 5 of the 6 searches. For Chronos-2-Small, the search reported in the paper is the lowest of the three, and the two other searches are above the frequency heuristic. The search is therefore not underperforming for this model: the reported search fell below the frequency heuristic by chance. In addition, the frequency heuristic itself is sensitive to the composition of the corpus: when we add downsampled data, it loses in all eight validation comparisons and up to 3.90 points of test skill score on
CB-II, whereas Auto-Chroma is far less sensitive (RQ1 and Appendix D.1, Figure 8).Confidence intervals. We computed confidence intervals with the procedure of fev-bench (Shchur et al., 2025): a paired bootstrap over the tasks of a benchmark, with 10,000 resamples. For each model and benchmark, we compare the Auto-Chroma portfolio directly with a heuristic portfolio on the test split, on the primary metric of the benchmark (MASE for
CB-II, SQL forfev-bench-mini). The skill score is one minus the geometric mean, over tasks, of the ratio between the error of Auto-Chroma and the error of the heuristic, in percent, so a positive value means that Auto-Chroma is better. Brackets give the 95% confidence interval. Bold marks a positive skill score.Auto-Chroma against the frequency heuristic:
CB-IICB-IICB-IIfev-bench-minifev-bench-minifev-bench-miniAuto-Chroma against the domain heuristic:
CB-IICB-IICB-IIfev-bench-minifev-bench-minifev-bench-miniAgainst the frequency heuristic, the skill score favors Auto-Chroma in 5 of 6 comparisons. The exception is Chronos-2-Small on
fev-bench-mini, the case that the repeated searches above cover. Against the domain heuristic, Auto-Chroma is better in all three comparisons onCB-II, by 2.49% to 4.67%. We will add these tables to the appendix of the final camera-ready version.W2/Q2: Contribution of the report-based memory
We ran two of the comparisons that the reviewer lists, on
fev-bench-mini: no search history and configurations and numerical scores without natural-language reports.(i) Without memory, the strategist is unreliable. In the zero-shot setting, the strategist has no search history: it sees only the two heuristic portfolios and their results, and it makes one proposal. We repeated this with 10 seeds for each model.
Validation skill score, SQL (%), against the frequency heuristic:
The proposals vary widely between seeds: for Chronos-2-Small, the standard deviation is about 40% of the mean. Even the best of the 10 proposals stays below Auto-Chroma for both models, and the mean is about half of it. The prior knowledge of the LLM alone is therefore not sufficient, and the memory is necessary.