acceptodds
Under review as a conference paper at ICLR 2027

AutoBricks: LLM-Driven Automated Brick Building for Spatio-Temporal Forecasting

Abstract

Designing high-performing spatio-temporal forecasters requires expertise in matching architectures to data topology, while off-the-shelf LLMs can introduce interface errors and tensor mismatches when generating forecasting code. We address this gap through retrieval-augmented modular assembly over a standardized brick registry, with a frozen LLM proposing architectures from task descriptions and retrieved experience. Our framework, AutoBricks, provides 38 operators with standardized invocation interfaces. A dual memory mechanism retrieves successful architectural configurations and failure trajectories to guide candidate generation, while execution and validation feedback inform subsequent proposals without updating LLM parameters. Experiments across 20 spatio-temporal datasets demonstrate high code executability and up to lower MAE than the strongest baselines. AutoBricks also substantially reduces search costs relative to NAS methods. Operator-pool and random-assembly comparisons further demonstrate the effectiveness of guided assembly. Code and demo are available at https://anonymous.4open.science/r/AutoBricks/.

Primary area:
learning on time series and dynamical systems
Keywords:
Spatio-Temporal Forecasting, LLM, Automated Data Model Construction, Neural Architecture Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y4aCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present an LLM-driven framework for automated spatio-temporal forecasting model design via retrieval-augmented modular assembly. In particular, a frozen LLM assembles architectures from a standardized registry of 38 bricks (aka operators) under a unified interface, guided by a dual memory of successful configurations and failure trajectories, and iteratively refined using execution diagnostics and validation-based “textual gradients.” Experiments on several datasets show good code executability, compared to the strongest baselines, as well as significantly reduced search cost relative to NAS methods.

Critical strengths

Automating ST architecture design with high executability and lowering entry barriers can be impactful for practitioners in transportation, urban computing, and environmental monitoring. This is relevant for ICLR community.

As methodological innovation, the standardized brick registry with a unified 4D input convention and a shared structural argument for grid and graph data seems an original way to me in order to reduce LLM interface errors / tensor mismatches. In addition, the dual memory mechanism proposed by the paper can store both positive (successful) architectural anchors and negative (failure) trajectories. This is novel in the domain and is operationalized with a concrete exclusion filter (with Jaccard similarity) to avoid incompatible combinations.

For experiments, the paper includes a broad evaluation across several datasets spanning grid and graph topologies. The proposed AutoBricks system is compared against a fair-enough suite of human-designed models, LLM-only generators, and automated search frameworks. Authors report code generation quality metrics and required ablations (without retrieval, negative constraints, feedback loop, brick registry), isolating contributions in a relevant way.

For clarity, the problem formulation is clear, and the system design is decomposed into infrastructure, retrieval/assembly, and execution/evaluation layers. Mechanisms like the dual memory filter, TF-IDF retrieval, and “textual gradients” are described to allow reproducibility.

Critical weaknesses

There are some technical limitations or concerns:

  • From a methodological point of view, the retrieval encoder is TF-IDF over textual serializations; despite the registry and constraints, structural/task alignment may be limited compared to more principled structural IRs (e.g., code-graph or FX-based representations). This may cap robustness as the memory grows or tasks diverge… Furthermore, the “hard feasibility constraint” in the objective is not fully formalized in practice. Actually the reward uses a cost penalty and OOM/shape errors produce negative feedback, but it is unclear to me what constraints are strictly enforced pre-training versus softly penalized, and how feasibility is checked beyond runtime OOM and shape validations?

  • From an experimental point of view, the backend selection is a potential advantage for the method: the main claims highlight that at least one AutoBricks backend wins per dataset. This cross-backend selection confers extra tuning capacity relative to baselines, and attribution of gains to specific components is therefore somewhat diffuse, even though per-backend results are included. The ablations use relatively small sample sizes; given the randomness in LLM generation and training, stronger statistical support for ablation conclusions would be required. Otherwise this deeply limit the conclusions of the authors...

  • For clarity, a few textual artifacts and formatting leftovers (e.g., duplicated phrases, partial captions) distract in places, and some hyperparameters are deferred to appendices not shown.

  • For positioning regarding the SOA, the paper compares to many ST baselines, but some strong recent backbones (e.g., COOL, MegaCRN) are discussed in related work but it is unclear if they appear in the registry or comparison tables beyond what’s shown?

Questions for authors

For main-table claims, are the AutoBricks results per dataset chosen from the best of multiple LLM backends or is each backend treated as a distinct method with fixed selection a priori?

How are training budgets controlled across baselines and AutoBricks?

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer y4aC (1/2)

We sincerely thank you for your careful reading and for recognizing the practical value of automated design and the methodological novelty of our work. Below we respond to each of your concerns.


W1. Retrieval encoder and hard feasibility constraints

The retrieved records are textual serializations of architecture descriptions and evaluation feedback, and the query is the task description (data shape, topology, forecasting window, and resource budget), so we adopt TF-IDF for its low overhead and inspectability. Structural compatibility is not judged by retrieval. The Jaccard filter over brick-name sets heuristically excludes candidates resembling previous structural failures, as it ignores ordering, connectivity, and hyperparameters (Appendix E.1), and the sandbox trial run then checks each instantiated candidate (Sections 3.1 and 3.3). In the warm-start setting with a larger memory (Appendix D, a 10-task source pool), the average number of convergence rounds on four target tasks drops from 9.75 to 5.5 and the final MAE decreases by 2.82% on average, showing that retrieval remains effective with multi-task memory at the evaluated scale. We will discuss structural representations such as code graphs or FX-based IRs as a natural extension.

Two checks are strictly enforced before training. The first is the exclusion filter in Eq. 4, which applies only to negative records with interface or topology incompatibilities, while records of OOM and poor predictive performance remain only in the retrieved context. The second is a sandbox forward pass after instantiation, which catches shape mismatches, interface errors, and OOM; candidates that fail do not enter training and receive a reward of −ρ-\rho (Appendix E.2). Since a forward pass does not cover the memory of backpropagation and optimizer states, an OOM during training is likewise recorded as a failed evaluation with reward −ρ-\rho, so the resource limit K\mathcal{K}, the memory of a single A100 80GB GPU, is enforced at both stages. Parameter count is not a hard constraint and affects ranking only through the soft penalty in Eq. 9, normalized and clipped with Pmax=10P_{max}=10M.

W2 & Q1. Cross-backend selection

AutoBricks is one framework whose LLM backend is a pluggable component fixed a priori in each run. We report four backends to show that it works with both locally deployed open-source models and API-based closed models, and to examine how LLM capability affects the search. Each backend is a separate row in Tables 2 and 3, and within each backend the final architecture is selected by validation MAE (Section 4.1). The statement in Section 4.2 that at least one backend performs best is a summary across these rows, and no main-table entry is selected across backends. To isolate this effect, the table below evaluates each backend on its own against the strongest non-AutoBricks method in each cell. This reference is itself the best of all baselines in that cell as measured on the test set, which is a wider selection than four backends. With a single backend fixed in advance, DeepSeek outperforms it in 29 of the 46 cells and ties in 2. When one backend is selected per dataset by validation MAE, AutoBricks outperforms it in all 46 cells without using test information, at about four times the search cost of a single backend.

BackendGrid (28 cells)Graph (18 cells)Total (46 cells)
Fixed LLaMA3.117 wins, 1 tie3 wins20 wins, 1 tie
Fixed Qwen2.514 wins, 1 tie12 wins26 wins, 1 tie
Fixed GPT-5-mini8 wins7 wins15 wins
Fixed DeepSeek16 wins13 wins, 2 ties29 wins, 2 ties
Selected by validation MAE28 wins18 wins46 wins

The attribution of gains does not rely on cross-backend selection, since every control fixes one backend per dataset. Under the same registry, search budget, and evaluation pipeline, AutoBricks-38 achieves MAE of 13.70 and 1.38 on TaxiBJ and PEMS-BAY, versus 16.35 and 2.18 for random assembly (Random-38, Table 8), so the gain comes from guided search rather than from the operator pool or the number of evaluated candidates. Under the same nine-brick pool, AutoBricks-9 also outperforms the adapted NAS baselines on six datasets by 1.1% to 12.7%, which separates the search mechanism from the operator pool. The ablations in Figure 5 likewise remove one component at a time under a fixed backend and dataset.

AuthorsOpenReview ↗

Official Response to Reviewer y4aC (2/2)

W2 & Q2. Tuning capacity, training budgets, and ablation statistics

The training budgets are as follows.

  • Hand-designed baselines and AutoBricks candidates share the same training setting, namely Adam, at most 100 epochs, and early stopping with patience 5 (Section 4.1, Appendix H.3). Baselines use official hyperparameters when available and are otherwise grid-tuned on validation over learning rate, batch size, and hidden dimension, with at most 18 configurations.
  • AutoBricks searches at most 20 rounds per task and backend and trains one checked candidate per round; failed candidates are not trained.
  • NAS baselines use the budgets recommended in their papers, typically 50 to 100 supernet epochs plus final retraining.
  • In the main tables, Codex and Claude Code use at most 10 repair rounds to obtain a runnable model; in the supplementary experiment, this is changed to at most 20 rounds returning the best validation model. Generated models use the same training setting as the other deep baselines.
  • Reported GPU hours include both search and final model retraining (Appendix H.3).

For executability, the 20 independent generation samples per variant (Section 4.5) yield significant differences. For Qwen on TaxiBJ, the full framework reaches Pass@1 of 20/20 (Wilson 95% CI 83.9% to 100%), whereas removing the brick registry yields 1/20 (0.9% to 23.6%); the intervals do not overlap, and Fisher’s exact test gives p ≈ 3×10⁻¹⁰. For accuracy, removing any of the four components increases MAE in all 16 combinations of variant and setting in Figure 5, which cover four backends and both topologies, with increases from 5.4% to 94.6%. For low-Pass@1 variants, MAE is computed only over executable samples, as noted in Section 4.5.

W3. Clarity and hyperparameters

Thank you for the writing suggestions. We will fix the duplicated phrases and incomplete captions. The hyperparameters mentioned are provided in Appendices B, E.1, E.5, and H.3 of the submitted PDF, and in the revision we will move the key ones into the main text, including K = 5, τ = 0.8, γ = 0.8, β = 0.5, and the 20-round budget.

W4. Comparison with recent backbones

MegaCRN is already included as a baseline in Table 3 and discussed in the related work. Neither MegaCRN nor COOL is a source model of the current brick registry. We thank you for pointing us to COOL, a well-designed model that jointly captures spatial and temporal relations from both prior and dynamic perspectives, and we will add it to the related work. COOL does not release official code, and its paper reports per-horizon results only on METR-LA, PEMS-BAY, PEMS07, and PEMS08. We attempted to reimplement it following the paper, but since several hyperparameters and architectural details are not specified, our results could not match those reported. To avoid an unreliable comparison, we did not include it in Table 3 and will explain this in Appendix H.1. We have contacted the first author by email to request the code. Once it is available, we will reproduce COOL with the authors’ code and parameters under our evaluation protocol and report the results.


Thank you again for your time and for the suggestions to improve our paper. We hope our responses address your questions, and we look forward to your further feedback. The revised version will be updated accordingly within the coming days.

I thank the authors for the clarity of their responses and the additional information provided. This allows me to better appreciate certain points, but it does not lead me to change my rating, which I consider fair given the overall criticisms raised regarding the submitted manuscript.

Dear Reviewer y4aC,

Thank you so much for getting back to us. We are glad to hear that our responses helped you better appreciate some of the points, which tells us the time we spent on the rebuttal were worthwhile. We have revised the paper based on your comments, including the points you raised on backend selection, the ablation statistics, and the hard constraint. The new version has now been uploaded. We fully respect your view, and if any further questions come up, please feel free to let us know at any time.

Thank you again for your time and your reply, and for seeing some strengths of AutoBricks. This means a lot to us and encourages us to keep pushing this work forward and we wish you all the best!

Best regards,

The Authors

Reviewer_YZaJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AutoBricks casts architecture design for spatio-temporal (ST) forecasting as retrieval-augmented modular assembly. Thirty-eight operators drawn from prior ST models are standardized to a common (B,T,N,C)(B, T, N, C) interface. A frozen LLM then proposes assembly configurations, conditioned on the task description, successful and failed configurations retrieved from a dual memory, and reflections on execution errors and validation results. Candidates are checked in a sandbox, trained, and scored by validation MAE with a parameter penalty over a 20-round search. On 20 grid and graph datasets with four LLM backends, the paper reports high executability and search costs of 1.2–1.4 GPU-hours. It also reports that, on every dataset, the best of the four backends is the best result in the table, with up to 15.1% lower MAE than AutoSTF on PEMS03.

Critical strengths

S1. The standardized brick registry is a useful, released artifact, and its role is convincingly demonstrated: removing it drops Pass@1 from 100% to 5% (Fig. 5).

S2. The evaluation is broad: 20 datasets, four backends, cost and sensitivity analyses, memory transfer, and operator-pool and random-assembly controls. The authors are also candid about the limits of attributing the gains (Sec. 4.2, App. H).

Critical weaknesses

W1. The headline accuracy relies on picking the best of four backends per dataset, and the largest gains are unexplained.

  • No single backend is consistently competitive. On PEMS03 the four backends range from 10.51 to 15.97 MAE, and the GPT-5-mini backend is worse than GWNet (14.65). On PEMS07, DeepSeek reaches 14.88 while the other three backends obtain 19.89–20.58.
  • Both the "up to 15.1%" claim and the significance test in App. A use this per-dataset selection.
  • The best PEMS03 and PEMS07 results are 27% and 22% below the best hand-designed model in the same table. Yet App. F.3 reports that about two-thirds of the final architectures are isomorphic to a source baseline. Gains of this size over those same baselines point to differences in the training and evaluation pipeline rather than to architecture discovery.
  • Three additions would resolve this:
    • report the discovered architectures;
    • retrain them alongside their source baselines in one shared pipeline;
    • report per-backend mean ± std.

W2. The baselines do not isolate the value of guided search.

  • Codex and Claude Code stop at the first runnable model and receive no validation feedback (App. H.2), whereas AutoBricks gets 20 rounds of validation-guided search. This explains why Claude Code is close to the Historical Average baseline on grid data (TaxiBJ MAE 27.72 vs. 29.82).
  • The agents should get the same train/validate budget, and existing LLM-based NAS methods (e.g., Xue et al., 2025; LLMatic; LLM4GNAS) should be included in the comparison.
  • The NAS baselines are restricted to 9 of the 38 operators.
  • Against the closest ST methods (AutoSTF, AutoST), the cost reduction is about 3.4–3.9×, not the headlined 18×/38×.
Questions for authors
  • Which architectures did each backend find on PEMS03 and PEMS07? Do the gaps remain when these architectures and their source baselines are retrained in one pipeline?
  • How should the backend be chosen for a new dataset? What do Tables 2–3 look like if the backend is chosen on validation MAE or fixed in advance?
  • Which numbers in Table 3 are reruns and which are published? Were AutoCTS and AutoSTF evaluated under exactly the same protocol as AutoBricks?
  • Can Codex and Claude Code be rerun with the same 20-round train/validate budget and with validation feedback? Do the GPU-hours in Fig. 4 include serving the local LLMs?
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer YZaJ (1/3)

We sincerely thank you for your time and careful review, and for recognizing some strengths of our work. We are also grateful that you specified concretely how your concerns could be resolved. Below we respond to and clarify each of them.


W1 & Q2. Cross-backend selection and backend choice for new datasets

We report four backends to show that the framework works with both locally deployed open-source models and API-based closed models, and to examine how LLM capability affects the search; results do differ across backends. Each backend is reported as a separate row in Tables 2 and 3, and within each backend the final architecture is selected by validation MAE (Section 4.1); the test set is not used in search or selection. For a new dataset, when the budget allows, one can search with all four backends and select by validation MAE, which outperforms the strongest baseline in all 46 cells at about four times the search cost of a single backend. Under a limited budget, we recommend fixing DeepSeek in advance, which wins 13 and ties 2 of the 18 graph cells. Both options are summarized below against the strongest non-AutoBricks method in each cell.

BackendGrid (28 cells)Graph (18 cells)Total (46 cells)MAE only (20 datasets)
Fixed LLaMA3.117 wins, 1 tie3 wins20 wins, 1 tie9 wins, 1 tie
Fixed Qwen2.514 wins, 1 tie12 wins26 wins, 1 tie9 wins, 1 tie
Fixed GPT-5-mini8 wins7 wins15 wins5 wins
Fixed DeepSeek16 wins13 wins, 2 ties29 wins, 2 ties11 wins, 1 tie
Selected by validation MAE28 wins18 wins46 wins20 wins

Under validation-based selection, the backends chosen for the six datasets in Appendix A coincide with those used in Appendix A (LLaMA3.1 for the three grid datasets, DeepSeek for PEMS03 and PEMS07, and Qwen2.5 for PEMS08). Therefore, the significance tests in Appendix A and the 15.1% result remain unchanged when backends are selected by validation MAE rather than by test performance. In Appendix A, each seed reruns the full search and training, and AutoSTF is also rerun with five seeds in the same pipeline.

W1 & Q1. Source of the larger gains on PEMS03 and PEMS07

The roughly two-thirds isomorphism rate in Appendix F.3 is a statistic over all tasks, and the two architectures on PEMS03 and PEMS07 are not among the isomorphic cases; their composition is given below. The isomorphism check also preserves only connectivity and brick types while ignoring scalar hyperparameters (Appendix F.3), so isomorphic architectures can still differ in settings such as hidden dimensions, which are themselves chosen by the search. Regarding the pipeline, all methods in Tables 2 and 3 run in the same pipeline with identical data splits, windows, Z-score normalization fitted on the training set, and zero masking.

Following your suggestion, the table below lists the architectures found by DeepSeek on PEMS03 and PEMS07. Results and search logs for all datasets and backends will be released in the anonymous repository.

DatasetSpatial brick × layersTemporal brick × layersFusionKey hyperparametersMAE
PEMS03SpatialAttention × 3WaveNet × 1ConcatSpatialAttention hidden dim 64, 4 heads, dropout 0.1; WaveNet residual and dilation channels 64, skip channels 128, kernel size 2, 3 blocks of 2 layers10.51
PEMS07ChebGCN × 1ResidualMLP × 3ConcatChebGCN order Ks = 3, Gaussian-kernel adjacency with symmetric normalization; ResidualMLP hidden dim 6414.88

The two architectures combine SpatialAttention from STTN with WaveNet from GWNet, and ChebGCN from STGCN with ResidualMLP from STID. The table below compares them, trained from scratch outside the search loop, with GWNet, STGCN, and STID, which contribute three of the four bricks, and with the nine-brick AutoSTF.

Model (5 seeds)PEMS03 MAEPEMS07 MAE
Found architecture, trained from scratch10.48 ± 0.4314.67 ± 0.53
GWNet14.63 ± 0.3620.63 ± 0.64
STGCN15.78 ± 0.4622.31 ± 0.62
STID15.56 ± 0.3919.62 ± 0.53
AutoSTF, 9-brick adapted12.37 ± 0.5816.72 ± 0.65

The per-backend mean ± standard deviation over the three search-and-training runs summarized in Table 3 is reported below.

BackendPEMS03 MAEPEMS07 MAE
LLaMA3.114.14 ± 0.4219.96 ± 0.52
Qwen2.512.55 ± 0.3719.89 ± 0.58
GPT-5-mini15.97 ± 0.4620.58 ± 0.55
DeepSeek10.51 ± 0.4414.88 ± 0.48
AuthorsOpenReview ↗

Official Response to Reviewer YZaJ (2/3)

W2 & Q4. Coding-agent baselines and LLM-based NAS

In the original setting, Codex and Claude Code stop at the first runnable model and receive no validation signal (Appendix H.2). Their code generation already involves internal trial and revision, whose number of iterations cannot be fixed externally, so we used the first runnable model, within at most 10 refinement rounds, as a well-defined stopping point. Following your suggestion, we changed the prompt to return the model with the best validation performance, allowed the agents to train and validate on their own, and set the maximum number of rounds to 20, matching AutoBricks. Under this setting, Claude Code reduces MAE on TaxiBJ and PEMS03 from 27.72 and 15.84 to 18.37 and 13.54, clearly better than Historical Average (29.82) on grid data, but still higher than AutoBricks (15.42 and 10.51), with about 1.7 times the GPU hours and 56 to 68 times the API cost.

For LLM-based NAS, we added LLMatic, a general code-level NAS method evaluated on CIFAR-10 and NAS-Bench-201 with about 2,000 candidates in the original paper. We replaced its seed networks with spatio-temporal models and its fitness with validation MAE, limited it to the same 20 evaluations, and used a locally deployed CodeGen-Mono 6.1B as the code generator. ISTS (Xue et al., 2025) is currently available only as a preprint, without a formally published version or official code, and a reproduction may differ substantially from the original, so we did not include it; we would be glad to reproduce it if you find it necessary. LLM4GNAS targets graph learning on static graphs, its search space contains neither temporal operators nor a multi-step forecasting pipeline, and it does not apply to grid data, so it was not included.

MethodSettingTaxiBJ MAEPEMS03 MAEGPU hours (TaxiBJ / PEMS03)API cost in USD (TaxiBJ / PEMS03)
Claude CodeOriginal, first runnable model27.7215.840.11 / 0.120.18 / 0.18
CodexOriginal, first runnable model30.1315.710.23 / 0.250.14 / 0.14
Claude CodeBest model, up to 20 rounds18.3713.542.1 / 2.41.68 / 2.04
CodexBest model, up to 20 rounds19.3514.242.4 / 2.81.45 / 1.78
LLMatic20 evaluations19.5813.273.8 / 4.70 / 0
AutoBricks (DeepSeek)20 rounds15.4210.511.2 / 1.40.03 / 0.03
AuthorsOpenReview ↗

Official Response to Reviewer YZaJ (3/3)

W2 & Q3. Operator pool and provenance of the NAS baselines

The NAS baselines in the main tables retain their own search algorithms and use topology-specific nine-brick subsets as candidate operators (Appendix H.3), because placing all 38 candidate operators in a supernet exceeds the memory of a single A100 80GB GPU in our implementation. All baselines in Tables 2 and 3, including AutoSTG, AutoCTS, and AutoSTF, were rerun by us in a unified pipeline with three seeds. No published numbers are used, and we will revise Appendix H.3 to state this explicitly. The NAS baselines share the data and evaluation protocol with AutoBricks and differ from it in search strategy, operator pool, and the search budgets recommended by their original papers.

The nine-brick subset does not weaken AutoCTS and AutoSTF. Compared with both methods rerun with official code in their original search spaces, the nine-brick adapted versions achieve 10.1% to 17.8% lower MAE on PEMS03, PEMS04, PEMS07, and PEMS08. Under the same nine-brick pool, AutoBricks-9 also outperforms both adapted methods on all four datasets, and together with TaxiBJ and PEMS-BAY in Table 8, it outperforms NAS-9 on all six datasets by 1.1% to 12.7%.

MethodSearch spacePEMS03PEMS04PEMS07PEMS08
AutoCTSOriginal14.9118.7419.9615.23
AutoCTS9-brick adapted12.6116.4017.9412.71
AutoSTFOriginal14.2117.7919.6914.68
AutoSTF9-brick adapted12.3815.7616.7212.06
AutoBricks (DeepSeek)9 bricks11.4815.3615.3211.93
AutoBricks (DeepSeek)38 bricks10.5115.3214.8811.55

W2 & Q4. Search cost

We would like to clarify that the paper does not present 18× or 38× as a headline result. The abstract states no ratio, Section 4.4 explicitly attributes the roughly 18× and 38× to DARTS and AutoSTG, and the conclusion reports a range of 3.4× to 38×, whose lower end is the 3.4× relative to AutoSTF; the ratio relative to AutoST is 3.9×, and AutoBricks also attains the lowest MAE on both datasets. GPU hours in Figure 4 include search and final model retraining (Appendix H.3), and search cost covers LLM inference, sandbox execution, and candidate training and validation (Appendix I). TaxiBJ uses the GPT-5-mini API. On METR-LA, the locally deployed Qwen2.5-7B takes 0.90 seconds per round on graph tasks (Table 4), about 18 seconds or 0.005 GPU hours over 20 rounds, which is less than 1% of the 1.2 GPU hours.


Thank you again for your time and review. We hope our responses and clarifications address your concerns, and please let us know if you have any further questions. We will reflect these updates in the revised version, which we plan to upload in the next few days.

Reviewer_Dsb9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents AutoBricks, a framework that designs spatio-temporal (ST) forecasting models by having an LLM assemble them from a fixed library of components, without updating the LLM's weights. Its parts are: (i) a registry of 38 ST operators ("bricks") taken from published grid and graph models, re-implemented or ported to PyTorch, with a shared 4D (B, T, N, C) interface and a structural argument G; (ii) a dual memory of successful configurations (M+) and failed ones (M-), retrieved by TF-IDF similarity. Retrieved structural failures also act as a hard exclusion filter, using Jaccard similarity over sets of brick names (Eq. 4); (iii) an epsilon-decaying mixture of random sampling and LLM proposals (Eq. 7); (iv) "dual-granularity" feedback. Sandbox execution errors and a validation reward (Eq. 9: inverse MAE minus a parameter-cost penalty) are summarised into reflective prompt updates (Eq. 10).

Critical strengths
  1. Practical, reusable engineering. Unifying 38 heterogeneous grid and graph operators behind one interface, including porting legacy TensorFlow and unreleased models (Sec. 3.1, Table 1), is useful to the community if the code is released as promised. It also explains the high executability: the LLM outputs a configuration, not free-form code.
  2. Broad empirical coverage: 20 datasets of both topologies, four backends, many human-designed, NAS and agentic-coding baselines, and supporting studies (Appendices A-D).
Critical weaknesses
  1. The headline comparisons with NAS methods are against restricted re-implementations. Appendix H says the adapted NAS baselines use topology-specific nine-brick subsets "following the memory limitations reported for our baseline implementations". The numbers in Table 8 match the main tables: NAS-9 on TaxiBJ is 16.78, the AutoST entry in Table 2; on PEMS-BAY it is 1.55, the AutoSTF entry in Table 3. So the AutoST/AutoSTG/AutoCTS/AutoSTF rows in Tables 2-3 appear to be these restricted versions. The abstract's "up to 15.1% lower MAE than the strongest baselines" (PEMS03, vs. AutoSTF) and the 3.4x-38x GPU-hour savings are therefore measured against handicapped versions of the methods, not the published ones. Table 8 is the only matched-pool control, and it covers two datasets, one backend and one run each.

  2. Picking the best backend per cell on test results inflates the main claim. Sec. 4.2 states that "at least one AutoBricks backend attains the lowest reported error for each dataset and metric, while no single backend dominates". This is a best-of-four selection made after seeing test metrics. Many grid margins over UniST are small, for example TrafficHZ MAE 0.234 vs. 0.236, Cellular 6.452 vs. 6.500 and TaxiNYC-I 5.571 vs. 5.650. Most of these wins would likely disappear for any single fixed backend; for example, in Table 2 GPT5-mini loses to UniST on Cellular, BikeNYC-I, TaxiNYC-I and others. The paper should report one pre-specified backend, or choose the backend on validation data. It should also report standard deviations for all main-table entries; currently only Appendix A has them, for 6 datasets.

Questions for authors
  1. Are the NAS rows in Tables 2-3 (AutoST, AutoSTG, AutoCTS, AutoSTF) the nine-brick restricted re-implementations? Can you add AutoSTF/AutoCTS with their original search spaces and published hyperparameters on at least PEMS03/04/07/08, or quote their published numbers under the same protocol?
  2. With a single backend fixed in advance (e.g., DeepSeek), how many dataset/metric cells does AutoBricks win in Tables 2-3?
  3. What architectures were found for PEMS03 and PEMS07? Does training them from scratch, outside the search loop, reproduce MAE about 10.5 and 14.9? How are zero or missing values handled in MAE/MAPE?
  4. Were the Appendix B sweeps scored on validation or test MAE? Were the defaults fixed before the main experiments?
2 comments
AuthorsOpenReview ↗

Official Response to Reviewer Dsb9 (1/2)

We sincerely thank you for recognizing the engineering value of the unified operator interface and the breadth of our experiments. The brick registry is publicly available in the anonymous repository, and we will continue to maintain it. We would also like to clarify that AutoBricks is not merely an engineering effort. It recasts open-ended code generation as retrieval-augmented assembly over a standardized registry, in which dual positive and negative memories and execution feedback guide a frozen LLM, reducing the manual effort of selecting spatio-temporal components and tuning hyperparameters. Below we respond to each of your concerns.


W1 & Q1. Operator pool of the NAS baselines

Yes, these rows are the nine-brick adapted versions described in Appendix H.3, which keep their own search mechanisms. We use these adaptations rather than the original implementations so that all NAS methods share the same brick implementations, data pipeline, and evaluation protocol as AutoBricks, and the comparison reflects the search strategy rather than differences in operator code or preprocessing. The pool is limited to nine bricks because a supernet over all 38 bricks exceeds the memory of a single A100 80GB GPU in our implementation, not to weaken these baselines.

The adapted AutoCTS and AutoSTF are not weakened; they are in fact stronger than the originals. We reran both methods with their official code, original search spaces, and published hyperparameters. Across all eight method and dataset pairs below, the nine-brick adapted versions achieve 10.1% to 17.8% lower MAE than the original versions. These NAS rows in the main tables are therefore strengthened rather than handicapped baselines, and they share the same brick implementations as AutoBricks. This also indicates that the registry-based adaptation can benefit existing NAS methods as a reusable operator pool.

We further extended the matched-pool control in Table 8 to these four datasets (DeepSeek, means over three seeds). Under the same nine-brick pool, AutoBricks-9 achieves lower MAE than both adapted NAS methods on all four datasets; together with TaxiBJ and PEMS-BAY in Table 8, it outperforms NAS-9 on all six datasets by 1.1% to 12.7%. It also outperforms Random-38, which uses all 38 bricks, on TaxiBJ, PEMS-BAY, PEMS03, and PEMS07 (16.35, 2.18, 12.24, and 16.46 for Random-38). Extending the pool to 38 bricks further reduces MAE by 0.3% to 8.4% on the four PEMS datasets. Relative to the original versions, AutoBricks (DeepSeek) reduces MAE by 13.9% to 29.5%, including 26.0% over the original AutoSTF on PEMS03.

MethodSearch spacePEMS03PEMS04PEMS07PEMS08
AutoCTSOriginal14.9118.7419.9615.23
AutoCTS9-brick adapted12.6116.4017.9412.71
AutoSTFOriginal14.2117.7919.6914.68
AutoSTF9-brick adapted12.3815.7616.7212.06
AutoBricks (DeepSeek)9 bricks11.4815.3615.3211.93
AutoBricks (DeepSeek)38 bricks10.5115.3214.8811.55

W2 & Q2. Cross-backend selection

Each backend is reported as a separate row in Tables 2 and 3, and within each backend the final architecture is selected by validation MAE (Section 4.1); the test set is not used in search or selection. The statement in Section 4.2 that at least one backend attains the lowest error summarizes the four backends. We report four backends to show that the framework works with both locally deployed open-source models and API-based closed models, and to examine how LLM capability affects the search.

With DeepSeek as the single backend for all datasets, AutoBricks wins 29 and ties 2 of the 46 cells. When one backend is selected per dataset by the mean validation MAE over three seeds, all 46 cells outperform the strongest non-AutoBricks method, so the conclusion of Section 4.2 holds without using test information. In practice, a single backend can be fixed in advance, selection by validation MAE can be used when the budget allows, or a backend router can be added before search.

BackendGrid (28 cells)Graph (18 cells)Total (46 cells)MAE only (20 datasets)
Fixed LLaMA3.117 wins, 1 tie3 wins20 wins, 1 tie9 wins, 1 tie
Fixed Qwen2.514 wins, 1 tie12 wins26 wins, 1 tie9 wins, 1 tie
Fixed GPT-5-mini8 wins7 wins15 wins5 wins
Fixed DeepSeek16 wins13 wins, 2 ties29 wins, 2 ties11 wins, 1 tie
Selected by validation MAE28 wins18 wins46 wins20 wins
AuthorsOpenReview ↗

Official Response to Reviewer Dsb9 (2/2)

W2 & Q2 (continued). Small margins on grid datasets

The three examples you listed, TrafficHZ, Cellular, and TaxiNYC-I, all come from LLaMA3.1. As a fixed backend, LLaMA3.1 wins 17 and ties 1 of the 28 grid cells, the most among single backends on grid tasks, so these results hold when this backend is fixed. Over five seeds, all three gaps are significant under a paired t-test (p < 0.05), where TrafficHZ is newly added following the setting of Appendix A.

Dataset (5 seeds)AutoBricks (LLaMA3.1)UniST
Cellular6.442 ± 0.0286.516 ± 0.031
TaxiNYC-I5.565 ± 0.0265.662 ± 0.039
TrafficHZ0.233 ± 0.0020.238 ± 0.003

Appendix A together with this addition covers both ends of the main tables. The graph datasets with the largest gains in Table 3 are PEMS03, PEMS07, and PEMS08 (15.1%, 11.0%, and 6.1%), and the four grid datasets with the smallest gaps in Table 2 are Cellular, TrafficHZ, TaxiNYC-II, and TaxiNYC-I (0.7% to 1.4%). Over five seeds, all of them pass the paired t-test (p < 0.01 for the graph datasets and p < 0.05 for the grid datasets). Due to space limits, the main tables report means over three seeds, and standard deviations for all AutoBricks entries will be added to the appendix.

Q3. Architectures found on PEMS03 and PEMS07 and their reproduction

The table below lists the architectures found by DeepSeek on PEMS03 and PEMS07. Results and search logs for all datasets and backends will be released in the anonymous repository. Users can reproduce them directly from the recorded configurations, or manually assemble their own models from the same bricks.

DatasetSpatial brick × layersTemporal brick × layersFusionKey hyperparametersMAE
PEMS03SpatialAttention × 3WaveNet × 1ConcatSpatialAttention hidden dim 64, 4 heads, dropout 0.1; WaveNet residual and dilation channels 64, skip channels 128, kernel size 2, 3 blocks of 2 layers10.51
PEMS07ChebGCN × 1ResidualMLP × 3ConcatChebGCN order Ks = 3, Gaussian-kernel adjacency with symmetric normalization; ResidualMLP hidden dim 6414.88

The two architectures combine SpatialAttention from STTN with WaveNet from GWNet, and ChebGCN from STGCN with ResidualMLP from STID, and neither is isomorphic to any source baseline (Appendix F.3). Trained from scratch outside the search loop over five seeds, they reach MAE of 10.48 ± 0.43 and 14.67 ± 0.53, consistent with 10.48 ± 0.45 and 14.68 ± 0.60 with search and with Table 3. Under the same pipeline, AutoSTF obtains 12.37 ± 0.58 and 16.72 ± 0.65. Missing values are linearly interpolated along the time axis during preprocessing (Appendix G.3), and zeros are masked in MAE and MAPE, consistent with traffic forecasting benchmarks since DCRNN and with BasicTS+.

Q4. Data split used in Appendix B

Appendix B reports test MAE. Values at the default settings match Tables 2 and 3, and candidate selection within each search uses only the validation set. The defaults were fixed before the main experiments based on design considerations, and Appendix B is a post hoc sensitivity analysis. With γ = 0.8 and ε_start = 1.0, the expected number of random explorations over 20 rounds is about 4.94 (Appendix E.5), roughly a quarter of the budget. With τ = 0.8, compact structures containing at most four distinct bricks are excluded only when their brick sets are identical (Appendix E.1). β = 0.5 imposes a bounded penalty on parameter count, equal to 0.5 at the 10M reference; since the scale of inverse MAE varies across tasks, the relative weight of this penalty differs across tasks (Appendix E.1). K = 5 retrieves at most five records from each of the positive and negative memories (Eq. 5), that is, at most ten of the at most 20 records per task under cold start, while keeping the prompt length manageable.


Thank you again for your time and for raising these detailed concerns. We hope our responses address them and that the additional results are worth your time. If any concerns remain, we would welcome further discussion. These results will be incorporated into our revised version in the coming days.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear SACs, ACs, and Reviewers,

We sincerely thank you for your careful reading, valuable suggestions, and patience. The revised paper has been uploaded, with all changes highlighted in blue. We note that two concerns were shared across the reviews. The first is whether reporting multiple LLM backends gives AutoBricks an additional selection advantage over the baselines, and the second is whether the comparison with the NAS baselines is fair. We address these two points first and then summarize the remaining revisions.

1. Choice of LLM backends

We report four backends to show that the framework works with both locally deployed open-source models and API-based closed-source models, and to examine how LLM capability affects the search. Each backend is a distinct method fixed in advance, reported as a separate row in Tables 2 and 3, and selects its final architecture by validation MAE without using the test set. The new Section 4.3 and Figure 3(a) quantify this.

  • Each cell is compared against the strongest non-AutoBricks method, which is itself selected from all baselines on test results.
  • Even so, DeepSeek alone, fixed in advance, wins 29 of the 46 cells and ties 2.
  • Selecting the backend per dataset by the mean validation MAE over three seeds wins all 46 cells without test information.
  • These validation-selected backends are exactly those used in Appendix A, so its significance tests and the 15.1% gain on PEMS03 are unaffected.

2. Comparison with NAS baselines

The NAS baselines (AutoST, AutoSTG, AutoCTS, and AutoSTF) keep their own search algorithms but use nine-brick subsets as candidate operators, so that they share the operator implementations, data pipeline, and evaluation protocol with AutoBricks. A supernet over all 38 bricks would exceed the memory of a single A100 80GB GPU.

  • The nine-brick versions of AutoCTS and AutoSTF achieve 10.1% to 17.8% lower MAE on PEMS03/04/07/08 than the originals rerun with official code, original search spaces, and published hyperparameters, so the adaptation does not weaken the baselines (Figure 3(b), Table 10).
  • With the same nine-brick pool, AutoBricks-9 outperforms the corresponding NAS method on all six datasets, with 1.1% to 12.7% lower MAE (Tables 10 and 11).
  • With the same 38 bricks and search budget, AutoBricks-38 reduces MAE by 9.6% to 36.7% relative to random assembly, so the gains come from guided search rather than a larger operator pool (Table 11).
  • The search cost savings are about 3.4× and 3.9× relative to the closest methods, AutoSTF and AutoST, while 18× and 38× refer to DARTS and AutoSTG (Section 4.5). Local LLM inference takes about 18 seconds per 20-round search, under 1% of the total GPU hours (Section 4.7).

3. Other revisions

  • The architectures found by DeepSeek on PEMS03 and PEMS07 reproduce their search MAE when retrained from scratch over five seeds in the same pipeline and remain below GWNet, STGCN, STID, and AutoSTF, and per-backend means and standard deviations are also reported (Figure 3(c), Table 7, Appendix F.3).
  • With validation feedback and 20 rounds, Codex and Claude Code improve substantially but remain less accurate than AutoBricks, and Claude Code incurs 56 to 68 times the API cost of AutoBricks. A comparison with LLMatic is also added (Figure 3(d), Table 9, Appendix H.2).
  • Removing any component increases MAE by 5.4% to 94.6% in all four backend and dataset settings of the ablation (Section 4.6).
  • The hard constraint in Eq. 2 is now defined as a GPU memory limit enforced through execution checks, while the parameter count enters only as a soft penalty in Eq. 9 (Section 2).
  • Key hyperparameters are moved into Section 4.1, and the appendices add five-seed results on TrafficHZ (Appendix A), clarify that the sensitivity analysis reports test MAE with defaults fixed before the main experiments (Appendix B), describe zero-value masking (Appendix G.3), and discuss structural representations for retrieval (Appendix I).
  • COOL is added to the related work, and Appendix H.1 explains why it is not included in the comparison.

Thank you again for your comments, which have made our paper more complete. If you have any remaining questions, we would be glad to discuss them further.

Best regards,

The Authors

AuthorsOpenReview ↗

A Kind Request for Further Discussion

Dear Reviewers,

Please allow us to draw your attention to this comment briefly. As the discussion period is drawing to a close, we would like to sincerely thank you for the valuable suggestions and thoughtful concerns raised during the review process, which have helped us reflect more deeply on our work. We are especially grateful to Reviewer y4aC for responding to our rebuttal. As we have further updated the manuscript to address the initial concerns, we would be grateful if you could take another look at the revised version.

Within the limited timeframe, we have conducted comprehensive additional experiments and manuscript revisions in hopes of resolving your concerns. Two key issues in the initial reviews, namely the choice of LLM backends and the setup of the NAS baselines, arose because our original design motivations were not communicated clearly enough, which led to misunderstandings of our setup. We have provided thorough clarifications on both, along with new experimental support. We hope these clarifications also make the core contribution of AutoBricks clearer, as it recasts open-ended model coding as retrieval-augmented assembly over a standardized brick registry, enabling a frozen LLM to design executable spatio-temporal models at low cost, and the released registry even strengthens existing NAS methods when used as their operator pool. If you still have any remaining questions, or if you would like us to provide any further experiments or analyses, we would greatly appreciate hearing from you before the discussion window closes, so that we still have an opportunity to respond. Should you feel that this work requires further improvement, please also let us know; we fully respect your perspective and will gladly incorporate your suggestions into our revision.

Regardless of the outcome, thank you very much for taking the time to carefully review our paper. Your feedback means a great deal to us, and we look forward to hearing from you.

With best wishes,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.