- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present an LLM-driven framework for automated spatio-temporal forecasting model design via retrieval-augmented modular assembly. In particular, a frozen LLM assembles architectures from a standardized registry of 38 bricks (aka operators) under a unified interface, guided by a dual memory of successful configurations and failure trajectories, and iteratively refined using execution diagnostics and validation-based “textual gradients.” Experiments on several datasets show good code executability, compared to the strongest baselines, as well as significantly reduced search cost relative to NAS methods.
Critical strengths
Automating ST architecture design with high executability and lowering entry barriers can be impactful for practitioners in transportation, urban computing, and environmental monitoring. This is relevant for ICLR community.
As methodological innovation, the standardized brick registry with a unified 4D input convention and a shared structural argument for grid and graph data seems an original way to me in order to reduce LLM interface errors / tensor mismatches. In addition, the dual memory mechanism proposed by the paper can store both positive (successful) architectural anchors and negative (failure) trajectories. This is novel in the domain and is operationalized with a concrete exclusion filter (with Jaccard similarity) to avoid incompatible combinations.
For experiments, the paper includes a broad evaluation across several datasets spanning grid and graph topologies. The proposed AutoBricks system is compared against a fair-enough suite of human-designed models, LLM-only generators, and automated search frameworks. Authors report code generation quality metrics and required ablations (without retrieval, negative constraints, feedback loop, brick registry), isolating contributions in a relevant way.
For clarity, the problem formulation is clear, and the system design is decomposed into infrastructure, retrieval/assembly, and execution/evaluation layers. Mechanisms like the dual memory filter, TF-IDF retrieval, and “textual gradients” are described to allow reproducibility.
Critical weaknesses
There are some technical limitations or concerns:
-
From a methodological point of view, the retrieval encoder is TF-IDF over textual serializations; despite the registry and constraints, structural/task alignment may be limited compared to more principled structural IRs (e.g., code-graph or FX-based representations). This may cap robustness as the memory grows or tasks diverge… Furthermore, the “hard feasibility constraint” in the objective is not fully formalized in practice. Actually the reward uses a cost penalty and OOM/shape errors produce negative feedback, but it is unclear to me what constraints are strictly enforced pre-training versus softly penalized, and how feasibility is checked beyond runtime OOM and shape validations?
-
From an experimental point of view, the backend selection is a potential advantage for the method: the main claims highlight that at least one AutoBricks backend wins per dataset. This cross-backend selection confers extra tuning capacity relative to baselines, and attribution of gains to specific components is therefore somewhat diffuse, even though per-backend results are included. The ablations use relatively small sample sizes; given the randomness in LLM generation and training, stronger statistical support for ablation conclusions would be required. Otherwise this deeply limit the conclusions of the authors...
-
For clarity, a few textual artifacts and formatting leftovers (e.g., duplicated phrases, partial captions) distract in places, and some hyperparameters are deferred to appendices not shown.
-
For positioning regarding the SOA, the paper compares to many ST baselines, but some strong recent backbones (e.g., COOL, MegaCRN) are discussed in related work but it is unclear if they appear in the registry or comparison tables beyond what’s shown?
Questions for authors
For main-table claims, are the AutoBricks results per dataset chosen from the best of multiple LLM backends or is each backend treated as a distinct method with fixed selection a priori?
How are training budgets controlled across baselines and AutoBricks?
4 comments
Official Response to Reviewer y4aC (2/2)
W2 & Q2. Tuning capacity, training budgets, and ablation statistics
The training budgets are as follows.
- Hand-designed baselines and AutoBricks candidates share the same training setting, namely Adam, at most 100 epochs, and early stopping with patience 5 (Section 4.1, Appendix H.3). Baselines use official hyperparameters when available and are otherwise grid-tuned on validation over learning rate, batch size, and hidden dimension, with at most 18 configurations.
- AutoBricks searches at most 20 rounds per task and backend and trains one checked candidate per round; failed candidates are not trained.
- NAS baselines use the budgets recommended in their papers, typically 50 to 100 supernet epochs plus final retraining.
- In the main tables, Codex and Claude Code use at most 10 repair rounds to obtain a runnable model; in the supplementary experiment, this is changed to at most 20 rounds returning the best validation model. Generated models use the same training setting as the other deep baselines.
- Reported GPU hours include both search and final model retraining (Appendix H.3).
For executability, the 20 independent generation samples per variant (Section 4.5) yield significant differences. For Qwen on TaxiBJ, the full framework reaches Pass@1 of 20/20 (Wilson 95% CI 83.9% to 100%), whereas removing the brick registry yields 1/20 (0.9% to 23.6%); the intervals do not overlap, and Fisher’s exact test gives p ≈ 3×10⁻¹⁰. For accuracy, removing any of the four components increases MAE in all 16 combinations of variant and setting in Figure 5, which cover four backends and both topologies, with increases from 5.4% to 94.6%. For low-Pass@1 variants, MAE is computed only over executable samples, as noted in Section 4.5.
W3. Clarity and hyperparameters
Thank you for the writing suggestions. We will fix the duplicated phrases and incomplete captions. The hyperparameters mentioned are provided in Appendices B, E.1, E.5, and H.3 of the submitted PDF, and in the revision we will move the key ones into the main text, including K = 5, τ = 0.8, γ = 0.8, β = 0.5, and the 20-round budget.
W4. Comparison with recent backbones
MegaCRN is already included as a baseline in Table 3 and discussed in the related work. Neither MegaCRN nor COOL is a source model of the current brick registry. We thank you for pointing us to COOL, a well-designed model that jointly captures spatial and temporal relations from both prior and dynamic perspectives, and we will add it to the related work. COOL does not release official code, and its paper reports per-horizon results only on METR-LA, PEMS-BAY, PEMS07, and PEMS08. We attempted to reimplement it following the paper, but since several hyperparameters and architectural details are not specified, our results could not match those reported. To avoid an unreliable comparison, we did not include it in Table 3 and will explain this in Appendix H.1. We have contacted the first author by email to request the code. Once it is available, we will reproduce COOL with the authors’ code and parameters under our evaluation protocol and report the results.
Thank you again for your time and for the suggestions to improve our paper. We hope our responses address your questions, and we look forward to your further feedback. The revised version will be updated accordingly within the coming days.
I thank the authors for the clarity of their responses and the additional information provided. This allows me to better appreciate certain points, but it does not lead me to change my rating, which I consider fair given the overall criticisms raised regarding the submitted manuscript.
Dear Reviewer y4aC,
Thank you so much for getting back to us. We are glad to hear that our responses helped you better appreciate some of the points, which tells us the time we spent on the rebuttal were worthwhile. We have revised the paper based on your comments, including the points you raised on backend selection, the ablation statistics, and the hard constraint. The new version has now been uploaded. We fully respect your view, and if any further questions come up, please feel free to let us know at any time.
Thank you again for your time and your reply, and for seeing some strengths of AutoBricks. This means a lot to us and encourages us to keep pushing this work forward and we wish you all the best!
Best regards,
The Authors
Official Response to Reviewer y4aC (1/2)
We sincerely thank you for your careful reading and for recognizing the practical value of automated design and the methodological novelty of our work. Below we respond to each of your concerns.
The retrieved records are textual serializations of architecture descriptions and evaluation feedback, and the query is the task description (data shape, topology, forecasting window, and resource budget), so we adopt TF-IDF for its low overhead and inspectability. Structural compatibility is not judged by retrieval. The Jaccard filter over brick-name sets heuristically excludes candidates resembling previous structural failures, as it ignores ordering, connectivity, and hyperparameters (Appendix E.1), and the sandbox trial run then checks each instantiated candidate (Sections 3.1 and 3.3). In the warm-start setting with a larger memory (Appendix D, a 10-task source pool), the average number of convergence rounds on four target tasks drops from 9.75 to 5.5 and the final MAE decreases by 2.82% on average, showing that retrieval remains effective with multi-task memory at the evaluated scale. We will discuss structural representations such as code graphs or FX-based IRs as a natural extension.
Two checks are strictly enforced before training. The first is the exclusion filter in Eq. 4, which applies only to negative records with interface or topology incompatibilities, while records of OOM and poor predictive performance remain only in the retrieved context. The second is a sandbox forward pass after instantiation, which catches shape mismatches, interface errors, and OOM; candidates that fail do not enter training and receive a reward of −ρ (Appendix E.2). Since a forward pass does not cover the memory of backpropagation and optimizer states, an OOM during training is likewise recorded as a failed evaluation with reward −ρ, so the resource limit K, the memory of a single A100 80GB GPU, is enforced at both stages. Parameter count is not a hard constraint and affects ranking only through the soft penalty in Eq. 9, normalized and clipped with Pmax=10M.
AutoBricks is one framework whose LLM backend is a pluggable component fixed a priori in each run. We report four backends to show that it works with both locally deployed open-source models and API-based closed models, and to examine how LLM capability affects the search. Each backend is a separate row in Tables 2 and 3, and within each backend the final architecture is selected by validation MAE (Section 4.1). The statement in Section 4.2 that at least one backend performs best is a summary across these rows, and no main-table entry is selected across backends. To isolate this effect, the table below evaluates each backend on its own against the strongest non-AutoBricks method in each cell. This reference is itself the best of all baselines in that cell as measured on the test set, which is a wider selection than four backends. With a single backend fixed in advance, DeepSeek outperforms it in 29 of the 46 cells and ties in 2. When one backend is selected per dataset by validation MAE, AutoBricks outperforms it in all 46 cells without using test information, at about four times the search cost of a single backend.
The attribution of gains does not rely on cross-backend selection, since every control fixes one backend per dataset. Under the same registry, search budget, and evaluation pipeline, AutoBricks-38 achieves MAE of 13.70 and 1.38 on TaxiBJ and PEMS-BAY, versus 16.35 and 2.18 for random assembly (Random-38, Table 8), so the gain comes from guided search rather than from the operator pool or the number of evaluated candidates. Under the same nine-brick pool, AutoBricks-9 also outperforms the adapted NAS baselines on six datasets by 1.1% to 12.7%, which separates the search mechanism from the operator pool. The ablations in Figure 5 likewise remove one component at a time under a fixed backend and dataset.