- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MOSAIC is an agentic system for automated model building, instantiated on financial time-series forecasting and generation. The pipeline (1) characterizes the task through VLM-based EDA; (2) retrieves relevant cases and selects top-k models from a curated bank (5 forecasting, 10 generation); (3) uses AST parsing to extract candidate modules, assembles them into a "blueprint", and prompts an LLM to produce a model that is checked by execution; and (4) refines the best candidate through a set of 39/49 discrete actions (e.g., adding normalization or changing the LR schedule), selected by an offline IQL policy using a 65-dim state and implemented through LLM code edits. The RL data comes from auto-generated tasks on a separate dataset bank, with failure rollback, trajectory branching and invalid-action masks. Across Crypto / LOB / Stock and four backbones, MOSAIC outperforms DS-Agent, ResearchAgent, TS-Agent and AutoGluon / Optuna on most reported metrics (−3% to −8% RMSE vs TS-Agent) and achieves 100% execution success; the evaluation also includes module-level comparisons, ablations and four non-financial datasets.
Critical strengths
-
The evaluation covers a breadth of dataset/backbone/task type settings. The reported execution success rate of 100%, compared with 20–80% for DS-Agent / ResearchAgent, suggests that the blueprint and verification loop improve reliability.
-
The paper uses a small discrete policy to select structured refinements and a frozen LLM to implement them, while treating rolled-back edits as negative supervision and masking invalid actions, offers a practical way to learn from costly and frequently unsuccessful code changes.
Critical weaknesses
I would view the proposed method as a reliable agentic workflow for a vertical set of data science tasks.
The broad framing rests on a relatively narrow implementation: "automated data science" is evaluated on financial time series with 15 hand-picked models, "architecture synthesis" combines modules from five forecasting models, and the RL component selects among 39 tuning actions after training on 4.5K transitions. The evaluation does not include a standard data-science / AutoML benchmark of the kind targeted by DS-Agent and AIDE.
Questions for authors
-
Are any case-bank entries derived from the Crypto (CTBench), LOB or Stock datasets used for evaluation?
-
Does the TS-Agent comparison use the same model bank, evaluation bank and LLM budget as MOSAIC? Which components are added or changed in MOSAIC relative to TS-Agent?
Thank you for the review. We respond to each point below.
W. The broad framing rests on a narrow implementation, with no standard data-science/AutoML benchmark.
We agree, and we will narrow the framing from automated data science to automated time-series modelling. This matches your view of MOSAIC as a reliable agentic workflow for a vertical set of tasks. Within that scope, the relevant comparisons are the agentic and AutoML baselines on time-series tasks in Tables 7–10, so we do not add a general data-science/AutoML benchmark.
Q1. Are any case-bank entries derived from the Crypto, LOB or Stock datasets used for evaluation?
No. The RL dataset bank is disjoint from the evaluation datasets at the asset level (Appendix A, Table 4). Case-bank entries are drawn from published financial benchmarks, and summaries of successful RL trajectories are added to the case bank (Appendix B).
Q2. Does the TS-Agent comparison use the same model bank, evaluation bank and LLM budget as MOSAIC? Which components are added or changed in MOSAIC relative to TS-Agent?