acceptodds
Under review as a conference paper at ICLR 2027

MOSAIC: Modular Orchestration for Structured Agentic Intelligence and Composition

Abstract

Automated data science is a structured model-selection problem: a solution must choose data transformations, feature representations, architecture, training procedure, evaluation protocol, and refinement strategy appropriate for a given task. AutoML systems automate parts of this process, but typically search within predefined pipeline, model, and hyperparameter spaces. LLM-based agents offer greater flexibility through retrieval, code generation, and execution feedback, yet their modelling decisions are often unstructured, difficult to verify, and hard to reuse across tasks. We introduce MOSAIC (Modular Orchestration for Structured Agentic Intelligence and Composition), a structured agentic framework for memory-grounded model selection and workflow construction. Given a task and dataset, MOSAIC builds a semantic task profile, retrieves relevant prior cases and source-code modules, and constructs a blueprint: an intermediate representation specifying the selected modelling components, their composition, interface constraints, and execution requirements. This blueprint turns model selection into a staged, context-grounded search process and grounds LLM-based code generation in retrieved evidence rather than unconstrained synthesis. Candidate models are validated by execution and refined using diagnostic feedback, training traces, task metrics, and a failure-aware reinforcement learning policy for long-horizon improvement. We instantiate MOSAIC on financial time-series forecasting and generation, a demanding application where models must satisfy predictive accuracy, distributional fidelity, execution reliability, and downstream financial criteria such as risk and tail behaviour. Experiments against AutoML and agentic baselines show that MOSAIC improves task performance, execution success, and decision traceability, demonstrating the value of treating automated data science as structured, reusable, and execution-grounded model selection. Our code is publicly available at https://anonymous.4open.science/r/MOSAIC-8778/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based agents, Automated data science, Agentic AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_x8FgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOSAIC is an agentic system for automated model building, instantiated on financial time-series forecasting and generation. The pipeline (1) characterizes the task through VLM-based EDA; (2) retrieves relevant cases and selects top-k models from a curated bank (5 forecasting, 10 generation); (3) uses AST parsing to extract candidate modules, assembles them into a "blueprint", and prompts an LLM to produce a model that is checked by execution; and (4) refines the best candidate through a set of 39/49 discrete actions (e.g., adding normalization or changing the LR schedule), selected by an offline IQL policy using a 65-dim state and implemented through LLM code edits. The RL data comes from auto-generated tasks on a separate dataset bank, with failure rollback, trajectory branching and invalid-action masks. Across Crypto / LOB / Stock and four backbones, MOSAIC outperforms DS-Agent, ResearchAgent, TS-Agent and AutoGluon / Optuna on most reported metrics (−3% to −8% RMSE vs TS-Agent) and achieves 100% execution success; the evaluation also includes module-level comparisons, ablations and four non-financial datasets.

Critical strengths
  1. The evaluation covers a breadth of dataset/backbone/task type settings. The reported execution success rate of 100%, compared with 20–80% for DS-Agent / ResearchAgent, suggests that the blueprint and verification loop improve reliability.

  2. The paper uses a small discrete policy to select structured refinements and a frozen LLM to implement them, while treating rolled-back edits as negative supervision and masking invalid actions, offers a practical way to learn from costly and frequently unsuccessful code changes.

Critical weaknesses

I would view the proposed method as a reliable agentic workflow for a vertical set of data science tasks.

The broad framing rests on a relatively narrow implementation: "automated data science" is evaluated on financial time series with 15 hand-picked models, "architecture synthesis" combines modules from five forecasting models, and the RL component selects among 39 tuning actions after training on 4.5K transitions. The evaluation does not include a standard data-science / AutoML benchmark of the kind targeted by DS-Agent and AIDE.

Questions for authors
  1. Are any case-bank entries derived from the Crypto (CTBench), LOB or Stock datasets used for evaluation?

  2. Does the TS-Agent comparison use the same model bank, evaluation bank and LLM budget as MOSAIC? Which components are added or changed in MOSAIC relative to TS-Agent?

1 comment

Thank you for the review. We respond to each point below.

W. The broad framing rests on a narrow implementation, with no standard data-science/AutoML benchmark.

We agree, and we will narrow the framing from automated data science to automated time-series modelling. This matches your view of MOSAIC as a reliable agentic workflow for a vertical set of tasks. Within that scope, the relevant comparisons are the agentic and AutoML baselines on time-series tasks in Tables 7–10, so we do not add a general data-science/AutoML benchmark.

Q1. Are any case-bank entries derived from the Crypto, LOB or Stock datasets used for evaluation?

No. The RL dataset bank is disjoint from the evaluation datasets at the asset level (Appendix A, Table 4). Case-bank entries are drawn from published financial benchmarks, and summaries of successful RL trajectories are added to the case bank (Appendix B).

Q2. Does the TS-Agent comparison use the same model bank, evaluation bank and LLM budget as MOSAIC? Which components are added or changed in MOSAIC relative to TS-Agent?

  • Shared setup. TS-Agent used the same model bank, the same evaluation bank and the same LLM backbones as MOSAIC. The LLM budget differs. MOSAIC makes additional LLM calls for EDA with VLM plot analysis and for model generation, so it uses more calls and tokens than TS-Agent. RL action selection itself makes no LLM calls, since the policy chooses each edit, and the frozen LLM only implements it. Per-method costs are in Table T-3 in response to Reviewer 2ZuD.
  • Added or changed components. MOSAIC keeps TS-Agent's case retrieval, model bank, refinement knowledge and two-phase optimisation with rollback. It adds semantic EDA with VLM plot analysis, blueprint-based generation from extracted code modules, and an offline-RL refinement policy trained with failure-aware branching. Our response to Reviewer 2ZuD (W2.1) gives the evidence for each.
Reviewer_w55UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOSAIC, an agentic framework for automated data science that formulates model selection and workflow construction as an evidence-grounded, staged procedure. Given a task and dataset, MOSAIC builds a semantic task profile, retrieves relevant prior cases and source-code modules, and constructs an intermediate blueprint that guides LLM-based code generation, execution, and iterative refinement. Experiments on financial time series forecasting and generation tasks show that MOSAIC improves both general time series and finance-specific performance metrics compared to AutoML and agentic baselines.

Critical strengths
  1. The methodology is well-motivated and well-structured. Generating a blueprint before letting an LLM synthesize a solution is a reasonable way to improve the robustness of candidate solutions, and separating high-level action selection (over a structured action space using an RL policy) from action execution (using an LLM to make code edits based on the selected structured action) is a reasonable design choice that makes the iterative refinement process more robust and structured.

  2. When learning the RL policy from offline trajectories, turning failed and degrading edits into useful supervision via trajectory branching and rollback is a generalizable idea that could benefit the training of a broader range of agents that rely on execute-and-refine loops, beyond data science agents.

  3. The evaluation on financial time series benchmarks is comprehensive, including both general time series and finance-specific metrics, as well as generalization to time series from other domains.

Critical weaknesses
  1. The paper introduces MOSAIC as an automated data science agentic framework. However, the evaluation is primarily conducted on financial time series forecasting and generation tasks (except for a few out-of-domain time series datasets). In MOSAIC, the case bank and the model bank also focus on forecasting and generation cases and models. Given the scope of the data and evaluation, it would be more appropriate to tone down the claim and position MOSAIC as a time series-specific data science agentic system.

  2. It is unclear which modules drive the gains of MOSAIC despite the ablation analysis of individual components. Comparing the results in Table 1 and Table 2, the full MOSAIC system appears to perform worse than ModelGen (MOSAIC) in several cases. For example, ModelGen (MOSAIC) achieves lower RMSE than MOSAIC on all three benchmarks including Stock, LOB, and Crypto. Does this suggest that the IQL component does not contribute to the final performance? While the detailed ablations within each individual module provide useful insights, it is also important to study how these modules interact with each other and contribute to the effectiveness of the complete system.

  3. Given the multiple stages and the LLM calls, it is unclear how computationally efficient MOSAIC is compared to the baselines, and whether its gains arise from the proposed design or from using more computational resources. Comparing to baselines in the number of LLM calls, or wall-clock time would provide a clearer picture of the efficiency of MOSAIC and further validate its effectiveness.

Questions for authors

A suggestion: it would be easier for readers to interpret the figures if the captions also included a brief summary of the main findings or intended takeaways. This would help readers quickly understand the purpose of each figure and connect it to the corresponding sections in the paper.

1 comment

Thank you for the clear review. We respond to each point below.

W1. The claims should be toned down to a time-series-specific system.

We agree that the current case bank, model bank, and empirical evaluation are specific to time series. We will therefore reposition MOSAIC as a framework for automated time-series model development, instantiated on financial forecasting and generation, and revise the abstract, introduction, and contributions accordingly. The non-financial forecasting results (Appendix H.8: macroeconomic, commodity, climate, and energy data) will be presented as evidence of generalisation across time-series domains, rather than data science tasks more broadly.

Nevertheless, the underlying methodology is not inherently restricted to time series. MOSAIC is designed to leverage users’ existing domain knowledge and resources, such as model banks and evaluation banks, to enhance task-solving capabilities through agentic model composition and refinement. These principles could potentially extend to other data science tasks, although such generalisation remains to be empirically validated.

W2. Which modules drive the gains? ModelGen (MOSAIC) has lower RMSE than full MOSAIC on all three benchmarks.

The full-MOSAIC rows were computed as w × ModelGen + (1 − w) × IQL where w is the win rate of ModelGen. The gap therefore compares RL-refined bank models with the ModelGen variant, and says nothing about how much RL contributes. What RL adds is shown at the module level. Refining retrieved models in isolation, IQL wins 83% of metric comparisons against LLM-only refinement (Figure 6, Tables 15–18). Removing its failure-aware components worsens results, and without trajectory branching, Crypto RMSE with GPT-4o rises from 0.209 to 0.219 (Tables 27–28).

W3. Computational efficiency compared with the baselines.

We add LLM calls, tokens and time for every baseline (See Table T-3 in response to Reviewer 2ZuD).

Suggestion: figure captions should state the main finding.

Thank you. We will add a one-line takeaway to every figure caption.

Reviewer_hdpgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MOSAIC, a structured agentic framework for automated model development that bridges fixed-search-space AutoML and unconstrained LLM code generation. It profiles each task and dataset, retrieves relevant prior cases and reusable code modules, and constructs an explicit blueprint specifying component selection, composition, interface constraints, and execution requirements before generating code. It further separates model refinement into high-level actions selected by an offline reinforcement-learning policy and concrete edits implemented by a frozen LLM, using rollback, trajectory branching, and invalid-action masking to learn from failed or degrading edits. The authors validate MOSAIC on financial time-series forecasting and generation across stock, limit-order-book, and cryptocurrency datasets, reporting strong overall results in predictive accuracy, distributional fidelity, execution reliability, and financial metrics against AutoML and agentic baselines. Ablations support the contributions of EDA and feature engineering, blueprint-guided generation, execution verification, and failure-aware refinement, while additional forecasting tasks suggest some generalization beyond the primary financial benchmarks.

Critical strengths

Strong overall empirical performance within the evaluated financial domain. Averaged across four LLM backbones, Table 1 shows that MOSAIC reduces forecasting RMSE relative to TS-Agent from 8.167 to 7.907 on Stock, from 0.157 to 0.144 on LOB, and from 0.220 to 0.210 on Crypto. For time-series generation, it reduces the LOB Marginal and Correlation distances from 0.870 to 0.587 and from 0.296 to 0.216, respectively, while reducing the Stock Marginal distance from 0.507 to 0.387. Table 2 further shows that the repository-grounded generated model is selected as the final output in 60–70% of forecasting and generation comparisons, compared with only 0–5% for naive LLM generation. Figure 6 also reports that IQL-based refinement wins 83% of metric comparisons and reaches the best candidate in 8.7 steps on average, versus 10.0 for LLM-only refinement. These results are consequential because they demonstrate practically meaningful improvements across multiple datasets, task types, metrics, and LLM backbones, providing substantive empirical support for the proposed structured pipeline.

Critical weaknesses

While the paper presents promising empirical results in the evaluated financial time-series settings, I highlight the following concerns, which I hope will help clarify its contribution.

  1. The methodological contribution is not sufficiently distinguished from recent modular and workflow-search systems. AgentSquare (Shang et al., ICLR 2025) explores module evolution and recombination, AFlow (Zhang et al., ICLR 2025) searches code-represented workflows using execution feedback, and ADAS (Hu et al., ICLR 2025) searches over executable agent programs. Although these methods address different tasks, they share several design principles with MOSAIC. MOSAIC’s failure-aware offline refinement is potentially useful, but it remains unclear whether the overall method provides a substantive algorithmic advance beyond integrating retrieval, modular composition, execution verification, and existing offline RL in a primarily financial setting. The paper’s baselines and ablations support the utility of its components, but do not fully resolve this novelty question.

  2. Dependence on the knowledge banks may limit genuinely open-ended architecture discovery. MOSAIC can create new combinations of retrieved components, but its effective search space remains shaped by the capabilities and refinement actions represented in its case, code, and refinement banks. The paper demonstrates recombination and idea transfer, but not whether MOSAIC can discover a useful modelling principle that is absent from those banks. Although they address different optimization problems, recent program-search methods provide a relevant contrast: A₂DEPT (Chen et al., ICML 2026) moves beyond fixed templates by evolving complete algorithms, while ShinkaEvolve (Lange et al., ICLR 2026) promotes open-ended exploration through code-novelty rejection sampling. Without a leave-one-family-out or mismatched-bank evaluation, it remains unclear whether MOSAIC’s gains reflect genuine discovery or favorable bank coverage, weakening its claimed advantage over fixed-search-space AutoML.

Shang, Y., et al. “AgentSquare: Automatic LLM Agent Search in Modular Design Space.” International Conference on Learning Representations (ICLR), 2025.

Zhang, J., et al. “AFlow: Automating Agentic Workflow Generation.” International Conference on Learning Representations (ICLR), 2025.

Hu, S., Lu, C., and Clune, J. “Automated Design of Agentic Systems.” International Conference on Learning Representations (ICLR), 2025.

Chen, B., et al. “A₂DEPT: Large Language Model–Driven Automated Algorithm Design via Evolutionary Program Trees.” International Conference on Machine Learning (ICML), 2026.

Lange, R., Imajuku, Y., and Cetin, E. “ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution.” International Conference on Learning Representations (ICLR), 2026.

Questions for authors
  1. Clarification and evidence regarding the methodological contribution: What capability is uniquely enabled by MOSAIC relative to modular and code-based workflow-search approaches such as AgentSquare, AFlow, and ADAS? The existing ablations show that removing MOSAIC’s components degrades performance, but they do not establish whether the blueprint representation or failure-aware offline-RL procedure provides a capability not already captured by generic modular workflow search. Could the authors provide a more explicit methodological mapping and, if feasible, an adapted comparison under a matched LLM backbone and execution budget, or explain why such an adaptation would not constitute a meaningful comparison? A clear distinction supported by controlled evidence would materially improve my assessment of the paper’s novelty.

  2. Clarification and evidence regarding open-ended architecture discovery: Can MOSAIC recover an effective modelling principle when a relevant model family, component, or refinement heuristic is deliberately absent from its knowledge banks, or is its discovery capability limited to recombining and adapting concepts already represented there? A controlled leave-one-family-out or held-out-principle experiment, reporting whether the generated architecture independently reconstructs or replaces the missing capability, would materially affect my assessment of the method’s generality and its claimed advantage over fixed-search-space AutoML. If such evidence is unavailable, clarifying whether MOSAIC claims open-ended discovery or only repository-grounded recombination would also materially affect my assessment.

1 comment

Thank you for the constructive review and for pointing us to closely related work in agent and program search. We respond to both points below.

W1/Q1. What does MOSAIC uniquely enable compared with AgentSquare, AFlow and ADAS?

We agree these systems share design principles with MOSAIC. AgentSquare recombines agent modules through standard interfaces, much as MOSAIC's blueprint composes model modules. AFlow refines code-represented LLM workflows with tree search and execution feedback, much as MOSAIC's refinement stage edits model code. ADAS has a meta-agent write new agents guided by an archive of past designs, a role MOSAIC's case bank also plays.

Three properties of model development keep these methods from applying directly, and MOSAIC is built around each:

  • Every evaluation requires training a model. AgentSquare, AFlow and ADAS score a candidate by running an LLM agent or workflow on validation questions. Here, every candidate must be trained first: in MOSAIC, warm-up training takes about 90% of a run's GPU time, and a run has 24 executions in total (Appendix G). Search strategies built on many cheap evaluations do not carry over. MOSAIC instead narrows the candidates through retrieval and a blueprint before anything is trained.
  • Their operators do not cover model code. Their search edits prompts, LLM-calling modules and control flow. Model development needs edits to architecture modules, tensor shapes, losses and training settings, where unconstrained code generation rarely works: naive LLM generation wins only 0–5% of comparisons in our experiments (Table 2). MOSAIC's blueprint fixes interfaces and shapes before code is written, and the generated model then wins 60–70% of comparisons.
  • Each search starts from scratch. All three run a separate search per task or domain and train nothing, keeping past attempts only as in-context notes within that search. With training-based evaluation, repeating this for every task is costly. MOSAIC carries experience across tasks: its refinement policy is trained offline on 83 forecasting and 75 generation tasks, including failed and degrading edits, and is applied frozen to new datasets, where it wins 83% of metric comparisons against LLM-only refinement (Figure 6).

For the same reasons, an adapted comparison would mean redefining their search spaces and operators over model code, at which point the result would no longer be the original method. For model code, the closest analogues are the program-search methods AlphaEvolve and AIDE, which we compare against at the module level with the same LLM backbones (Figure 5, Tables 11–14).

W2/Q2. Open-ended discovery or repository-grounded recombination?

We would place MOSAIC between pure repository-grounded recombination and open-ended discovery. The backbone and any transplanted components must come from the banks. MOSAIC does not invent a new model family from nothing, and its search is bounded by what the banks cover, as our Limitations paragraph notes. The code that connects and adapts these components, however, is written by the LLM and is not restricted to the banks. This includes adapters, conditioning modules, output heads and loss terms. We will also discuss A₂DEPT and ShinkaEvolve, which pursue open-ended discovery through sustained evolutionary search. MOSAIC's new code is instead written within a single blueprint and draws on the LLM's general modelling knowledge.

Both case studies in Appendix H.9 contain such components, none of which belongs to any shortlisted model:

  • Crypto forecasting (Case 1, GPT-4o). The generated model combines DLinear's series decomposition, TimeMixer's multi-scale convolutions and PatchTST's attention. The LLM adds a gated residual head that blends the network output with the most recent observation through a learned gate. The model reaches an RMSE of 0.205, vs 0.217 for the best single model (PatchTST).

  • LOB generation (Case 2, Claude Opus 4). The generated model uses Diffusion-TS as the backbone and FIDE's frequency-inflated score block as a direct component. It reinterprets COSCI-GAN's per-channel coordination as a new conditioning module inside the diffusion network. It also adds a tail-aware loss weighting, motivated by heavy tails that EDA flagged. The model reaches a Marginal distance of 0.513, vs 0.767 for Diffusion-TS alone.

Reviewer_2ZuDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MOSAIC, an LLM-based framework for automated model development, evaluated on financial time-series forecasting and generation. MOSAIC first analyzes the dataset using statistical EDA and VLM analysis of plots. It then retrieves similar tasks and selects three candidate models from a model bank. The system extracts reusable code modules and builds a blueprint that specifies how they should be combined. An LLM uses the blueprint to generate a new model, which competes with the retrieved candidates, and the best candidate is then refined. The framework also introduces an offline RL policy for model refinement. The policy uses IQL to select high-level editing actions, while a frozen LLM implements the changes. Failed edits are recorded through trajectory branching, rollback, and invalid-action masking. Experiments on three financial datasets and four LLM backbones show improvements over existing AutoML and agentic baselines.

Critical strengths
  1. The framework has a clear and practical design. MOSAIC separates model selection, architecture planning, code generation, and refinement into distinct stages. The blueprint provides explicit module interfaces and shape constraints, making generated models easier to inspect and debug. The refinement design is also sensible. Instead of training an LLM to edit code directly, the authors use offline RL to select high-level actions and leave implementation to a frozen LLM. The failure-aware mechanism makes use of unsuccessful experiments rather than simply discarding them.

  2. The paper provides a fairly extensive evaluation. The experiments cover forecasting and generation across three datasets and four LLM backbones. The authors compare the model-generation module with AlphaEvolve, AIDE, EffiLearner, and LLM4EFFI, and the refinement policy with BC, DQN, and LLM-only refinement. The paper includes component ablations, and the appendix provides per-backbone results and examples of generated architectures. The RL training tasks also use a separate dataset bank with disjoint assets, which helps reduce the risk of direct data leakage.

Critical weaknesses
  1. The results do not clearly support the claimed benefits of the RL component, and they are hard to check. The first row of Table 2, labelled ModelGen and defined in Section 4.1 as model generation without RL, outperforms the full MOSAIC row of Table 1 on most of the averaged metrics. For example, ModelGen achieves a LOB marginal distance of 0.547 versus 0.587 for full MOSAIC, and a Crypto ΔSharpe of 6.23 versus 7.97. According to Section 4.1, full MOSAIC uses RL refinement only when the generated model is not selected as the winner. This raises a basic question: if RL improves refinement, why does adding it often make the full system worse? This is also difficult to reconcile with the module-level results, where IQL outperforms LLM-only refinement. The paper attributes improved execution reliability to RL, but TS-Agent also achieves a 100% success rate without RL. Moreover, the main results lack standard deviations or confidence intervals, so it is unclear whether this gap, or the smaller improvements, exceed run-to-run variation. Some reported results also need clarification. For example, the same IQL configuration appears to have different values in Tables 17 and 29, while removing invalid-action masking increases Steps to Best Incumbent by exactly 0.40 in 23 of 24 comparisons in Tables 27 to 30. The results are also hard to verify because the main-text figures are too small to read.

  2. The novelty and generality of the framework are limited, and the evaluation does not fully justify its complexity. Much of MOSAIC builds on existing methods. TS-Agent already provides case retrieval, a model bank, refinement knowledge, and a two-phase optimization workflow with rollback. MOSAIC mainly adds EDA/VLM analysis, blueprint-based generation, and an offline IQL policy. These are useful additions, but the individual techniques are largely established. Related work such as AutoML-Agent, published at ICML 2025, also deserves discussion. More importantly, the paper presents MOSAIC as a general framework for automated data science, yet the main experiments use only financial time series. The model bank contains only five forecasting and ten generation models. New architectures are mainly generated by combining modules or ideas from three retrieved candidates, while RL selects from only 39 or 49 predefined actions. It is unclear whether this approach offers substantial advantages over simpler search methods within the same limited space, or whether it generalizes beyond these settings. Appendix G reports the cost of a single MOSAIC run, about 43 minutes and 70 LLM calls on Stock with GPT-5.4, but gives no costs for the baselines and no comparison under matched computational budgets. Without these comparisons, it is difficult to separate the benefits of the proposed method from those of additional search.

Questions for authors
  1. Why is RL used only when the generated model loses? Since ModelGen without RL beats full MOSAIC on most metrics, what does RL add? Are the ModelGen rows in Tables 2, 11 to 14 and 23 to 26 the no-RL variant?

  2. Since each result is averaged over five runs, can you report the standard deviation across those runs? Why does the same IQL setting have different values in Tables 17 and 29? Why does removing invalid-action masking add exactly 0.40 steps in 23 of 24 cases in Tables 27 to 30?

  3. How does MOSAIC compare with a simple last-value forecast? How were AutoGluon and Optuna set up? Can you report the cost of the baselines and compare all methods under the same compute budget?

  4. Beyond TS-Agent, how much does MOSAIC gain over a simple search in the same model bank, for example tuning each bank model with standard hyperparameter search and picking the best on validation? Can the Appendix H.8 experiment be extended to generation tasks and more backbones?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2ZuD (Part 1)

Thank you for the detailed review. Your points led to several corrections, set out below.

W1.1 Full MOSAIC trails ModelGen.

The full-MOSAIC rows were computed from the ModelGen and IQL rows. The ModelGen row is the outcome when the generated model wins, and the IQL row is the outcome when a retrieved model wins. The two rows describe different scenarios. The generated model wins 60–80% of comparisons against retrieved models (Table 2), while the IQL row refines retrieved models only. They cannot be compared directly, and mixing them pulls full MOSAIC toward the IQL row. Specifically, full MOSAIC was computed as w × ModelGen + (1 − w) × IQL where w is the win rate of ModelGen. The gap to ModelGen is therefore (1 − w) × (IQL − ModelGen): a comparison between RL-refined bank models and the ModelGen variant. It says nothing about whether RL refines better or worse than the LLM.

W1.2 What RL adds.

The module-level results answer this independently of the formula. Refining retrieved models in isolation, IQL wins 83% of metric comparisons against LLM-only refinement (Figure 6, Tables 15–18). The RL ablations also show that each failure-aware component contributes. Without soft revert, Stock RMSE with GPT-4o rises from 8.013 to 8.252, and Crypto ΔSharpe from 13.524 to 15.385. Without trajectory branching, Crypto RMSE with GPT-4o rises from 0.209 to 0.219 (Tables 27–28). These results show that IQL refines retrieved models better than the LLM alone.

W1.3 Execution reliability.

We agree. Execution success cannot show RL's contribution, because TS-Agent also reaches 100% (Tables 7, 9, 10). We attribute the 100% success rate to blueprint-guided generation and execution verification, and will remove the sentence in Section 4.2 that credits RL for it. RL's advantage lies instead in the quality of its refinement decisions. It picks edits that improve retrieved models more often than LLM-only refinement does. Its failure-aware training (rollback, trajectory branching and invalid-action masking) is designed to avoid repeating edits that failed or degraded a model, and removing these components worsens results (Tables 27–28, also see W1.2 What RL adds).

W1.4 Standard deviations and inconsistent numbers in Tables 17 and 27–30.

  • We report mean ± std in Table T-1 for the system-level comparison (Table 7) over 5 seeds from a re-run with seeds shared across methods.

T-1. Representative system-level performance (mean ± standard deviation) across LLM backbones

Forecasting

DatasetMetric ↓MOSAICTS-AgentDS-AgentResearchAgentAutoGluon
StockRMSE8.0974 ± 0.40988.2763 ± 0.63738.5650 ± 0.42099.3549 ± 0.40178.5790 ± 0.3476
LOBRMSE0.1426 ± 0.00440.1557 ± 0.00650.1814 ± 0.00550.2012 ± 0.00640.2210 ± 0.0086
CryptoRMSE0.2039 ± 0.03470.2162 ± 0.04170.2390 ± 0.03570.2603 ± 0.04160.2379 ± 0.0476
CryptoΔSharpe7.9675 ± 2.74929.4596 ± 3.598932.8812 ± 7.345914.4467 ± 4.707822.6427 ± 9.7303

Generation

DatasetMetric ↓MOSAICTS-AgentDS-AgentResearchAgentOptuna
StockCorrelation0.3488 ± 0.06510.3921 ± 0.09240.4836 ± 0.06920.3806 ± 0.08040.5075 ± 0.0076
LOBCorrelation0.2096 ± 0.06290.2977 ± 0.09260.4025 ± 0.07220.3157 ± 0.06710.3465 ± 0.1312
CryptoΔES0.0925 ± 0.01950.0908 ± 0.02580.1398 ± 0.02950.1147 ± 0.02180.1231 ± 0.1333
  • Tables 17 vs 29. Both tables report valid results for the same IQL configuration, but they were obtained from separate experimental runs using different random seeds. We will unify the seed sets and update both tables to report consistent results.

  • Identical Steps values. These come from a bug in retrieving Steps values. When a logged value was not found, the retrieval code returned a fallback value instead. The corrected values after-rerun, are in the Table T-2. Invalid-action masking reduces Steps to Best Incumbent in 15 of 24 dataset–backbone–task comparisons, with an average reduction from 5.73 to 5.39 steps. The effect is small and not consistent across cells, and we will revise Appendix H.7 accordingly.

T-2. Corrected effect of invalid-action masking on Steps to Best Incumbent (IQL+mask / no-mask)

DatasetGPT-4o FGPT-5.4 FClaude FNova FGPT-4o GGPT-5.4 GClaude GNova G
Stock4.6 / 2.63.6 / 5.25.6 / 4.44.8 / 7.05.2 / 6.46.2 / 4.28.2 / 4.66.6 / 7.4
LOB6.8 / 8.01.4 / 6.44.4 / 4.86.2 / 9.64.0 / 5.67.8 / 3.44.8 / 6.66.6 / 7.0
Crypto6.4 / 5.83.0 / 3.87.6 / 6.46.2 / 7.06.6 / 8.23.4 / 3.42.8 / 3.46.6 / 6.4
AuthorsOpenReview ↗

Response to Reviewer 2ZuD (Part 2)

W1.5 Main-text figures are too small to read.

We will move an averaged system-level results table, with standard deviations, into the main text. We will also split Figures 3–6 into larger panels and add a one-line takeaway to each caption.

W2.1 Relation to TS-Agent.

MOSAIC keeps TS-Agent's case retrieval, model bank, refinement knowledge and two-phase optimisation with rollback. It adds three components, each going beyond what TS-Agent can do:

  • New models, not only bank models. TS-Agent selects and refines models from its bank. MOSAIC extracts code modules from the retrieved models and composes a new model through a blueprint with explicit interface and shape constraints. The generated model beats the retrieved bank candidates in 60–80% of runs (Table 2). Without the blueprint, that drops to 0–20%, and LOB RMSE worsens by up to 52% (Claude: 0.095 → 0.145; Appendix H.6).

  • Refinement that learns from failures. MOSAIC's offline RL policy is trained on past refinement trajectories, including failed and degrading edits. Refining retrieved models, it wins 83% of metric comparisons against LLM-only refinement (Figure 6), and removing its failure-aware components worsens results (Tables 27–28).

  • Data profiling before modelling. Semantic EDA with VLM plot analysis characterises the data before model selection and generation. In Case 2 (Appendix H.9), it flagged heavy tails, which led to a tail-aware loss weighting in the generated model.

W2.2 AutoML-Agent.

AutoML-Agent is a multi-agent LLM framework for full-pipeline AutoML, uses retrieval-augmented planning over external knowledge, splits each plan into sub-tasks for specialised data and model agents, and verifies plans in stages. MOSAIC differs in three ways:

  • Execution-grounded search. AutoML-Agent evaluates candidate plans training-free, by prompting the LLM to predict their outcomes, and runs code only for the selected plan. MOSAIC trains and evaluates every candidate during warm-up.
  • Code-grounded architecture composition. AutoML-Agent's model agent searches, tunes and ranks candidate models. MOSAIC extracts source-code modules from retrieved models and composes new architectures through a blueprint with explicit interface and shape constraints (W2.1).
  • Learning across runs. AutoML-Agent revises plans from failures within a run, but does not describe a learned policy or memory across runs. MOSAIC keeps a case bank of past tasks and trains an offline RL refinement policy on trajectories from 83 forecasting and 75 generation tasks, including failed and degrading edits.

W2.3 Scope.

We agree that the evaluation supports a narrower claim than "automated data science". We will reposition MOSAIC as a framework for automated time-series modelling, instantiated on financial forecasting and generation.

W2.4 Limited model bank and action space, and comparison with simpler search.

We emphasize that MOSAIC is designed to leverage existing model banks provided by users, rather than relying on a predefined or comprehensive collection of models. Accordingly, the model banks used in our experiments are intended to be illustrative rather than exhaustive, demonstrating how MOSAIC can effectively compose and refine models already available to users.

The action space is small by design. Its 17 structural operations and 22 (forecasting) or 32 (generation) hyperparameter bins were chosen from the refinements that past cases in the case bank show to have the largest effect. It therefore covers the changes that matter most for the bank models RL refines. Applying RL to models outside the bank, including generated ones, is left for future work (see Q1). The bank bounds the backbones MOSAIC can draw on, but the generated model goes beyond the individual bank models and beats them in 60–80% of runs (W2.1).

On simpler search: our AutoML baselines are the search you describe. Both use MOSAIC's model bank: AutoGluon for forecasting (the 5 bank models) and Optuna for generation (the 10 bank models). Each selects on the same validation split as MOSAIC. Averaged over backbones, the generated model outperforms this search on 24 of 25 metrics.

W2.5 Cost and matched budgets.

See per-method costs and a comparison at a matched budget in Table T-3. On Stock forecasting with GPT-5.4, MOSAIC incurs higher computational costs than the agentic baselines, but maintains relatively modest peak RAM usage.

T-3. Computational cost comparison (Stock forecasting, GPT-5.4)

MethodTotal LLM tokens ↓LLM calls ↓Running time (min) ↓Peak RAM (MB) ↓
MOSAIC416,61170432,240.4
TS-Agent221,22835282,943.4
DS-Agent227,87732172,393.1
ResearchAgent188,8564181,409.2
AutoGluon001020,501.5
AuthorsOpenReview ↗

Response to Reviewer 2ZuD (Part 3)

Q1. Why is RL used only when the generated model loses? What does RL add? Are the ModelGen rows the no-RL variant?

  • Our offline trajectories were collected on retrieved bank models only. Newly generated architectures therefore fall outside the data the policy was trained on, and the policy may not transfer to them. Extending RL to generated models, which are not known in advance, is left for future work.

  • What RL adds. See W1.1 and W1.2.

  • Are the ModelGen rows the no-RL variant? Yes. As defined in Section 4.1, ModelGen (MOSAIC) refines with the LLM and uses no RL. The ModelGen rows in Tables 11–14 and 23–26 come from the same runs, and Table 2 reports their averages over backbones.

Q2. Standard deviations, Tables 17 vs 29, and the 0.40 step gap.

See W1.4.

Q3. Last-value forecast, the AutoGluon and Optuna setup, and baseline costs.

  • Last-value forecast. See results in Table T-4. Averaged over backbones, Mosaic beats it on all reported forecasting metrics.

T-4. Forecasting comparison with the last-value baseline, Mosaic results copied from Table 1.

DatasetMethodRMSE ↓MAE ↓MAPE (%) ↓sMAPE (%) ↓
StockMOSAIC7.9074.8532.0112.004
StockLast-value8.4525.2192.1012.113
LOBMOSAIC0.1440.1070.0480.048
LOBLast-value0.1670.1130.0510.051
CryptoMOSAIC0.2100.0521.5481.553
CryptoLast-value0.2250.0551.5681.569
  • AutoGluon and Optuna. AutoGluon and Optuna do not use an LLM, so each has one row rather than one per backbone. Both AutoGluon and Optuna use the same task-specific model pools as MOSAIC. Their setup: AutoGluon uses RMSE-based model selection and a default fitting budget of 3,600 seconds per dataset/run. The Optuna runner uses TPE sampling (seed=42) with a default budget of 100 trials per study. Its predefined search-space template includes learning rate 10−510^{-5}–10−210^{-2} (log scale), batch size {16, 32, 64, 128}, model dimension {32, 64, 128, 256, 512}, and dropout 0–0.5 (step 0.05).

  • Baseline costs and matched budgets. See W2.5.

Q4. Simple search in the same bank, and extending Appendix H.8.

  • Simple search. See W2.4.

  • Appendix H.8. Appendix H.8 currently covers forecasting on four non-financial datasets (macroeconomic, commodity, climate and energy) with GPT-5.4. We could not extend it to generation tasks and further backbones within the discussion period.

Thank you for the detailed response. It addresses my concerns about the overstated scope and the novelty compared with TS-Agent. The authors also admit several mistakes and inaccurate conclusions, for example that the 100% execution success rate does not come from RL, and the bug behind the 0.40 step difference. The authors also promise to improve the readability of the figures.

However, several of my concerns remain unresolved.

  1. The response says the full MOSAIC results are calculated by the equation w × ModelGen + (1 − w) × IQL, but Section 4.1 says that Full MOSAIC runs the complete system. So the main results were not obtained from actual runs of the full system.

  2. The contribution of RL is still not shown. The new standard deviations show that most of MOSAIC's lead over TS-Agent is within one standard deviation, and MOSAIC is even worse than TS-Agent on Crypto ΔES.

  3. The comparison is not fair enough. On Stock with GPT-5.4, MOSAIC uses about 1.9 times the tokens, 2 times the LLM calls and 1.5 times the running time of TS-Agent, but there is no comparison under a matched budget. Also, except for TS-Agent, the baselines are weaker than the last-value forecast on all three datasets.

Therefore, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.