acceptodds
Under review as a conference paper at ICLR 2027

Neural Architecture Search with LLM-Discovered Fusion Operators for Multimodal Classification

Abstract

Fusion operators play an important role in multimodal classification (MMC) by determining how representations from different modalities interact and are fused, thereby affecting the achievable performance. However, the fusion operators used in existing MMC methods are largely human-designed, limiting the diversity of fusion mechanisms that can be explored.We propose LLMDFO-NAS, a neural architecture search framework that automatically constructs and explores the fusion operator space using Large Language Models (LLMs) and evolutionary search.Specifically, it leverages LLMs to generate executable and validated fusion primitives and employs evolutionary search to discover effective primitive compositions as candidate fusion operators.Candidate fusion operators are efficiently evaluated across multiple architecture contexts using an architecture-aggregated zero-cost proxy with Pareto selection, and the high quality ones among them are selected as the discovered operators.These discovered operators then define the operator library for downstream multimodal architecture search, forming a two-level automated design process. Experiments on five representative benchmarks show that the discovered operator spaces consistently improve the resulting fusion architectures across different modality configurations.Our work extends automated design from architecture optimization within a fixed functional space to the joint design of functional and structural spaces.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal Fusion, Neural Architecture Search, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LfF8OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes LLMDFO-NAS, a framework for multimodal classification that integrates fusion operator discovery with neural architecture search (NAS). Unlike traditional multimodal NAS methods, which search for network architectures within a fixed library of fusion operators, this approach first leverages a large language model (LLM) to generate fusion primitives. These primitives undergo rigorous validation to ensure their viability in terms of execution, shape compatibility, gradient propagation, and determinism. Subsequently, these primitives are combined into complex fusion operators via evolutionary search; during this process, a "zero-cost" proxy metric, based on the expressive capacity of an operator within various architectural contexts, is employed alongside a Pareto selection mechanism to filter the generated operators. The resulting library of operators is then utilized in the subsequent search for multimodal fusion architectures, effectively expanding the search space to encompass both fusion functionality and network architecture. The proposed method is evaluated against various existing multimodal NAS approaches across four major benchmarks. Additionally, supplementary experiments are conducted on the MV-RML2016.10A dataset, focusing on the classification of wireless signals under low signal-to-noise ratio (SNR) conditions.

Critical strengths
  1. Expanding the scope of the search space offers notable methodological value. Traditional multimodal Neural Architecture Search (NAS) typically treats fusion operators as predefined functional units, focusing primarily on the connections, interaction sequences, and overall architectural structure. In contrast, this paper represents the fusion operators themselves as executable programs, employing a combination of Large Language Models (LLMs) and evolutionary search to automatically construct these operator combinations.

  2. The experiments encompass a diverse range of multimodal configurations, with preliminary validation provided for proxy metrics. The paper’s four primary benchmarks cover various input configurations, including image-text, multi-view visual data with labels, RGB-depth, and video-skeleton data. Furthermore, the authors report a strong correlation on the EgoGesture dataset between the rankings derived from zero-cost proxies and the validation performance achieved after full training. This evidence suggests that the authors have not only proposed a framework for searching fusion operators but have also sought to verify the predictive validity of their core selection mechanism.

Critical weaknesses
  1. Certain aspects of the problem formulation and experimental protocols are insufficient to fully substantiate the validity of the results. 1.1: There is an inconsistency between the multi-label task definition for the MM-IMDB dataset and the format of the model's outputs that remains to be addressed. 1.2: The normalization axes for the operators described in Appendix F5 may not align with the functions claimed in the paper.

  2. While the authors conduct ablation studies on the LLM backbone and the evolutionary composition strategy, and report on the correlation between zero-cost proxies and training performance, the current evidence fails to definitively answer the following key questions: Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition? Furthermore, given a fixed budget for evaluating candidate solutions, do zero-cost proxies perform better than random selection?

Questions for authors
  1. Certain aspects of the problem formulation and experimental protocols are insufficient to fully substantiate the validity of the results. 1.1: There is an inconsistency between the multi-label task definition for the MM-IMDB dataset and the format of the model's outputs that remains to be addressed. 1.2: The normalization axes for the operators described in Appendix F5 may not align with the functions claimed in the paper.

  2. While the authors conduct ablation studies on the LLM backbone and the evolutionary composition strategy, and report on the correlation between zero-cost proxies and training performance, the current evidence fails to definitively answer the following key questions: Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition? Furthermore, given a fixed budget for evaluating candidate solutions, do zero-cost proxies perform better than random selection?

  3. The paper constructs zero-cost proxies using random Gaussian inputs and covariance spectral entropy, and incorporates primitive complexity c(o)c(o) into the Pareto selection process. Please provide a precise numerical definition for c(o)c(o), explain how these values ​​are assigned, and clarify whether they are generated by the LLM itself.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LfF8

We thank the reviewer for the constructive feedback, which will help improve the quality of our manuscript.

Q1. Multi-label prediction on MM-IMDB and normalization axes in F5.

Certain aspects of the problem formulation and experimental protocols are insufficient to fully substantiate the validity of the results. 1.1: There is an inconsistency between the multi-label task definition for the MM-IMDB dataset and the format of the model's outputs that remains to be addressed. 1.2: The normalization axes for the operators described in Appendix F5 may not align with the functions claimed in the paper.

We sincerely apologize for these oversights and will clarify the relevant parts in the revised manuscript. About 1.1: The softmax shown applies to single-label tasks. For MM-IMDB, we instead use 23 independent logits, and trained with weighted binary cross-entropy. So each genre is predicted independently. We used these predictions to compute the weighted F1 reported in Table 1. About 1.2: The axis=0 takes the maximum across views in the first operation. Once the view dimension is removed, axis=0 refers to the batch dimension. The second operation uses the current batch to compute a mean and variance for each feature and standardizes each feature separately. It means that our method perform maximum aggregation across views, followed by normalization of each feature across the batch. We will clarify them in the revised manuscript.

Q2. Guided Fusion Operator Discovery vs. Random Composition

While the authors conduct ablation studies on the LLM backbone and the evolutionary composition strategy, and report on the correlation between zero-cost proxies and training performance, the current evidence fails to definitively answer the following key questions: Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition? Furthermore, given a fixed budget for evaluating candidate solutions, do zero-cost proxies perform better than random selection?

We thank the reviewer for insightful comments, which motivate us to further analyse the role of LLMs in our method. Both methods used the same library of 15 LLM-generated primitives. For the random baseline, we constructed five fusion operators by randomly selecting and connecting two to four primitive nodes per operator. We evaluated both five-operator libraries using the same downstream NAS procedure on NUS-WIDE-128. The result shows that LLM-guided composition produced a more effective fusion library from the same primitives in this experiment.

MethodNUS-WIDE-128 ACC(%)
Random composition76.08%
LLMDFO-NAS76.69%

To evaluate the effectiveness of the zero-cost proxy, we ranked 200 held-out candidate compositions according to their zero-cost proxy scores and validation accuracy after training on the other three datasets. The results are reported below.

DataSpearman correlationKendall correlation
NUS-WIDE-1280.94500.7856
EgoGesture0.91030.7413
NTU RGB-D0.95000.7985

The two rankings agreed closely across all three benchmarks. We further compared proxy selection with random selection under different evaluation budgets.

DatakProxy ACC(%)Random ACC(%, Mean±SD)
NUS-WIDE1076.303371.4929±3.9311
NUS-WIDE2076.259671.2767±2.8249
NUS-WIDE5076.230871.3354±1.6231
Ego1095.187995.0264±0.0633
Ego2095.184995.0246±0.0448
Ego5095.166295.0240±0.0259
NTU1091.694090.7996±0.5575
NTU2091.697490.8099±0.3658
NTU5091.643190.8577±0.1998

For each budget, we selected the top-k candidates ranked by proxy scores and calculated their mean accuracy. Random selection uniformly sampled k candidates without replacement from the same pool and averaged their accuracies. This process was repeated 1000 times, with results reported as mean ± SD. Proxy selection consistently achieved higher mean accuracy than random selection across all three datasets, especially on NUS-WIDE. It demonstrates the effectiveness of the zero-cost proxy in identifying promising candidate compositions.

Q3. Primitive complexity c(o)

The paper constructs zero-cost proxies using random Gaussian inputs and covariance spectral entropy, and incorporates primitive complexity (c(o)) into the Pareto selection process. Please provide a precise numerical definition for (c(o)), explain how these values ​​are assigned, and clarify whether they are generated by the LLM itself.

During primitive generation, the LLM sees the fusion task, the allowed tensor operations, and the implementation constraints. It generates the code for each primitive and uses its own judgment to estimate how complex the resulting computation is. The prompt sets the score range and leaves the scoring criteria to the LLM. Our code checks the range and stores the returned score as a fixed cost for that primitive.

Thank you very much for your rebuttal. However, my concerns remain unaddressed, so I will keep my score unchanged.

We sincerely apologize that our previous response may not have fully addressed your concerns. We greatly appreciate your constructive feedback. To better address your concerns and provide stronger supporting evidence, we have conducted additional experiments and further analysis.

Q2.1. Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition?

We thank the reviewer for this insightful comment, which motivate us to further analyse the role of LLMs in our method. We compare three strategies using the same library of 15 LLM-generated primitives on NUS-WIDE-128. The results are reported below.

MethodNUS-WIDE-128 ACC(%)
Random composition76.08%
Conventional evolutionary composition76.19%
LLMDFO-NAS76.69%

Notably, conventional evolutionary composition achieves only a small improvement of 0.11 percentage points over random composition. It means that conventional evolutionary operations provide limited benefits in exploring primitive combinations. In contrast, LLMDFO-NAS achieves improvements of 0.61 and 0.50 percentage points over random and conventional evolutionary composition, respectively. It means that LLM guidance provides more effective direction for exploring the composition space. These results further support the effectiveness of LLM-based evolutionary composition.

About Q2.2: We sincerely apologize for the incorrect data in our previous response. This error was caused by an incorrect parameter setting in the code. We noticed this issue during a subsequent check after submitting our response. However, we received your feedback while we were in the process of correcting the data. The corrected results have now been updated in our previous response. We sincerely regret this oversight and apologize for any confusion and inconvenience caused, as well as for taking up your valuable time.

Q3. The paper constructs zero-cost proxies using random Gaussian inputs and covariance spectral entropy, and incorporates primitive complexity (c(o)) into the Pareto selection process. Please provide a precise numerical definition for (c(o)), explain how these values ​​are assigned, and clarify whether they are generated by the LLM itself.

During primitive generation, the LLM is provided with the fusion task, the allowed tensor operations, and the implementation constraints. It generates the executable code for each primitive and assigns a complexity score (c(o)) based on its own judgment. The prompt specifies the scoring range [0, 4] and leaves the scoring criteria to the LLM. Our code checks the range and stores the returned score as a fixed cost for that primitive. During evolutionary search, the complexity of a composition is calculated as the sum of the costs of its constituent primitives.

Finally, we would like to further clarify our motivation. The size and quality of the search space affect the potential performance of NAS. Our motivation is to leverage the generative capabilities of LLMs to create reusable fusion primitives and expand the search space. A key challenge is how to make effective use of LLMs to generate primitives. We further employ LLM-guided evolutionary search to explore primitive compositions. Notably, our experiments demonstrate that LLMDFO-NAS can discover effective primitive combinations beyond those manual designs. We will further clarify this motivation in the manuscript. We hope these revisions adequately address your concerns. Please feel free to let us know if you have any further questions or concerns. We sincerely thank you again for your time and valuable suggestions.

Reviewer_SVxbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a neural architecture search framework for automatically constructing and exploring the fusion operature space by leveraging LLM instead of adopting manually designed ones. Specifically, LLM is used to generate some primitives and evolutionary search is used to obtain the compositions of the primitives as fusion operators. Five benchmarks are adopted to evaluate the performance with some performance gain.

Critical strengths
  1. Fusion operators represented as executable programs, instead of selecting from a fixed operator library.
  2. The three identified challenges are valid ones.
Critical weaknesses
  1. Limited originality and impact
  • The general ideas have been proposed for general-purpose algorithms and heuristics, and this work mainly applies it to multimodal learning.
  • The primitives being considered to form fusion operators exclude parameterized modules such as attention, gating and bilinear interactions (as stated in the paper).
  • The efficient evaluation of the candidate solution is based on an existing proxy - zero-cost fitness screening in MBTF-NAS (cited in the paper).
  1. Experimental results are not particularly impressive
  • The performance gain is small and the standard deviation is not reported. It seems like a sophisticated NAS search method being used for a not so-complicated problem.
  • The primitives being learned are not particularly special and a few more in fact could be hand-drafted to support the NAS, which in turn may explain why the performance improvement is not substantial.
Questions for authors

Question 1: As mentioned in the paper, the primitives being considered to form fusion operators exclude parameterized modules such as attention, gating and bilinear interactions. What is the implication? Will these restrictions affect the overall representation power of the possible fusion operators?

Question 2: LLM generates the initial population. Are they valid computational graphs? Should the evolutionary process be designed to ensure validity as far as possible to reduce the need to discard too many during the evolutionary search?

Question 3: A search over J randomly sampled multimodal fusion architecture is involve. How sensitive is the overall results given different pools of fusion architectures being considered and given value of J?

Question 4: A complexity descriptor is also introduced for the guiding the search. How sensitive is the overall results given different designs of the complexity descriptor? How optimal is the descriptor's design?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SVxb (1/2)

Thank you for noting that our paper is clear and easy to follow, and for raising these questions. We clarify what LLMDFO-NAS adds to multimodal NAS and report additional comparisons with an expanded hand-designed operator library, repeated searches, and sensitivity analyses.

W1. Originality and scope of the contribution

Limited originality and impact

The general ideas have been proposed for general-purpose algorithms and heuristics, and this work mainly applies it to multimodal learning.

The primitives being considered to form fusion operators exclude parameterized modules such as attention, gating and bilinear interactions (as stated in the paper).

The efficient evaluation of the candidate solution is based on an existing proxy - zero-cost fitness screening in MBTF-NAS (cited in the paper).

We thank the reviewer for these constructive comments, which helped us better assess the contribution of our approach to fusion operator discovery. MBTF-NAS searches for architectures using a predefined fusion operator library. Building on prior work in LLM-guided evolutionary search, LLMDFO-NAS automatically discovers executable fusion primitives and explores how to compose them into operators for downstream NAS. For example, one discovered primitive measures how neighboring feature dimensions co-variation across views and adds a bounded correction to the view mean. It preserves the output shape and introduces no trainable parameters. Its implementation is shown below.

import tensorflow as tf
def primitive(inputs, fused_nb_feats):
    x = tf.stack(inputs, axis=0)
    mean = tf.reduce_mean(x, axis=0)
    centered = x - mean
    covariance = tf.reduce_mean(
        centered * tf.roll(centered, shift=1, axis=-1), axis=0
    )
    return mean + tf.tanh(covariance)

For operator selection, we reuse the expressivity proxy from MBTF-NAS and aggregate its scores across sampled architectures. This lets us compare candidate compositions in several architectural contexts without fully training a model for each candidate or fitting operator-specific weights.

W2. LLMDFO-NAS shows consistent gains across runs and outperforms an expanded hand-designed library

Experimental results are not particularly impressive

The performance gain is small and the standard deviation is not reported. It seems like a sophisticated NAS search method being used for a not so-complicated problem.

The primitives being learned are not particularly special and a few more in fact could be hand-drafted to support the NAS, which in turn may explain why the performance improvement is not substantial.

We thank the reviewer for raising these concerns, which helped us assess the reliability of our results and the practical value of the discovered operators. We ran downstream NAS three times each on MM-IMDB and EgoGesture. The corresponding 95% confidence intervals were [64.65%, 64.85%] for weighted F1 on MM-IMDB and [95.30%, 95.41%] for accuracy on EgoGesture. Table 2 also reports three runs per LLM backbone on NUS-WIDE-128. The gains over MBTF-NAS in Table 1 are modest and comparable in magnitude to those achieved by earlier methods. On EgoGesture, our gain was 0.08 percentage points, compared with 0.04 points for MBTF-NAS over CSG-NAS. On MM-IMDB, our 0.27-point gain was close to the 0.28-point gain of CSG-NAS over CoMO-NAS.

Dataset and metricRun 1 (%)Run 2 (%)Run 3 (%)Mean ± SD (%)95% confidence interval (%)Published MBTF-NAS (%)
MM-IMDB, weighted F164.8064.7364.7364.75 ± 0.0464.65–64.8564.53
EgoGesture, accuracy95.3895.3495.3595.36 ± 0.0295.30–95.4195.29

On NTU RGB-D, we compared three fusion libraries using the same data split and downstream NAS budget. The hand-designed library included sum, mean, max, elementwise product, and trimmed mean, among other operators.

Fusion library used for downstream NASNumber of operatorsAccuracy (%)
Hand-designed operators1591.24
LLM-generated primitives used directly1591.67
LLM-discovered compositions592.06

At the same library size, the LLM-generated primitives improved accuracy by 0.43 percentage points over the hand-designed operators. Searching over compositions yielded a further gain of 0.39 percentage points with a library of five operators.

AuthorsOpenReview ↗

Response to Reviewer SVxb (2/2)

Q1. Parameter-free fusion

As mentioned in the paper, the primitives being considered to form fusion operators exclude parameterized modules such as attention, gating and bilinear interactions. What is the implication? Will these restrictions affect the overall representation power of the possible fusion operators?

The restriction excludes learned attention projections, trainable gates, and learned bilinear weights. The LLM can still discover parameter-free attention, such as weighting input views by their feature norms. This simplifies evaluation across architectures because there are no operator-specific weights to fit.

Q2. Validity of the generated graphs

LLM generates the initial population. Are they valid computational graphs? Should the evolutionary process be designed to ensure validity as far as possible to reduce the need to discard too many during the evolutionary search?

During initialization, recombination, and mutation, the LLM proposes graphs using validated primitives. The framework checks that nodes reference only original inputs or earlier nodes and that every node contributes to the output. It then compiles the graph before evaluation.

Q3. Size of the architecture pool

A search over J randomly sampled multimodal fusion architecture is involve. How sensitive is the overall results given different pools of fusion architectures being considered and given value of J?

A small J makes operator scores more sensitive to the sampled architectures, while increasing J reduces sampling noise. The 0.07 percentage-point difference between J = 32 and J = 64 may reflect variation in the quality of operators discovered across runs. Given the similar accuracies, we use J = 32 to reduce evaluation cost.

Number of contexts JNUS-WIDE-128 accuracy(%)
1676.47
3276.69
6476.62

Q4. Choice of complexity measure

A complexity descriptor is also introduced for the guiding the search. How sensitive is the overall results given different designs of the complexity descriptor? How optimal is the descriptor's design?

Two fusion graphs can have the same number of nodes and still differ in complexity. Our design captures this difference by assigning different costs to different primitives. It achieved the highest accuracy in the ablation, showing that accounting for which primitives a graph uses helps guide the search.

Complexity measureNUS-WIDE-128 accuracy(%)
Our method76.69
Count the nodes76.17

Thanks for the clarification. My evaluation of the paper stays the same after considering also the responses during the rebuttal.

We are sorry that our responses have not been sufficient to change your assessment. If you still have any concerns that have not been adequately addressed, please let us know. We are more than willing to make every effort to address them. Thank you again for your time and valuable feedback.

Reviewer_hWFrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a neural architecture search framework for multimodal classification, called LLMDFO-NAS. In particular, the proposed framework consists of two levels of search: fusion operator discovery and multimodal architecture search. In the first stage, this work uses LLMs to generate executable fusion primitives and evolutionary search to discover effective operator compositions. An architecture-aggregated zero-cost proxy is further introduced to evaluate candidate operators without full training. In the second stage, the discovered operators are used to construct the search space for multimodal architectures. Finally, experiments on 5 benchmarks are conducted to demonstrate the effectiveness of the proposed framework. In general, I believe this work is interesting, but I still have some questions for the authors for clarification.

Critical strengths
  1. The idea is intuitive and interesting. Instead of searching multimodal architectures based on predefined fusion operators, this work further explores the automatic discovery of fusion operators, enriching the design space for multimodal learning.

  2. The proposed two-level search framework is reasonable. Combining LLM-based primitive generation with evolutionary search provides a flexible way to explore different fusion strategies. Besides, the architecture-aggregated zero-cost evaluation helps reduce the computational cost of operator selection.

  3. The main results seem promising. The proposed framework achieves competitive performance across different multimodal classification benchmarks. The authors also provide several ablation studies to investigate the effectiveness of different components.

Critical weaknesses
  1. I have some questions about the necessity of LLMs in the proposed framework. Based on Table 4, the discovered operators mainly consist of relatively simple operations, such as max aggregation, mean aggregation, and feature normalization. These operations can also be manually defined. What is the fundamental advantage of using LLMs to discover these operators? Would a larger manually designed operator library achieve similar performance under the same search budget? I think this comparison is critical for demonstrating the main contribution of this work.

  2. Based on Table 1, the improvements over MBTF-NAS are relatively limited on some benchmarks. For example, the improvements on MM-IMDB and EgoGesture are only 0.27 and 0.08 percentage points, respectively. Are these improvements statistically significant? Besides, the search time increases from 0.41 to 2.33 hours on MM-IMDB and from 4.12 to 13.87 hours on NTU RGB-D. I wonder whether these improvements justify the additional search cost. A more detailed performance-efficiency analysis would strengthen the practical contribution.

  3. This paper uses an architecture-aggregated zero-cost proxy to evaluate the discovered fusion operators. However, the proxy mainly measures representational expressivity based on randomly initialized networks and Gaussian inputs. Does higher expressivity necessarily indicate better multimodal fusion? Although Figure 3 shows a strong correlation on EgoGesture, would this relationship remain consistent across other benchmarks and modality configurations? I think this is important for supporting the generalizability of the proposed evaluation strategy.

  4. Some experimental settings need further clarification. First, the proposed framework contains both operator discovery and architecture search, but the individual contribution of each stage is not sufficiently isolated. Would evolutionary composition search without LLM-generated primitives achieve similar results? Second, Table 5 shows that F5 accounts for 66.4% of the operator usage, while F4 contributes only 0.8%. Does this suggest that only a small subset of discovered operators is actually necessary? I think further analysis would help clarify the effectiveness of the learned operator library.

Questions for authors
  1. What is the fundamental advantage of using LLM-generated fusion primitives instead of a larger manually designed operator library?

  2. Are the relatively small improvements on MM-IMDB and EgoGesture statistically significant across multiple independent search runs?

  3. How does the architecture-aggregated zero-cost proxy correlate with downstream performance on benchmarks other than EgoGesture?

  4. What is the individual contribution of LLM-based primitive discovery compared with evolutionary composition search?

  5. Since F5 dominates the discovered operator usage, would using only the top one or two operators achieve comparable performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer hWFr (1/2)

We thank the reviewer for the constructive feedback and for noting that our paper is clear and easy to follow. We address the questions below with additional experimental results.

Q1 / W1. The advantage of LLM-guided operator discovery.

I have some questions about the necessity of LLMs in the proposed framework. Based on Table 4, the discovered operators mainly consist of relatively simple operations, such as max aggregation, mean aggregation, and feature normalization. These operations can also be manually defined. What is the fundamental advantage of using LLMs to discover these operators? Would a larger manually designed operator library achieve similar performance under the same search budget? I think this comparison is critical for demonstrating the main contribution of this work.

What is the fundamental advantage of using LLM-generated fusion primitives instead of a larger manually designed operator library?

We sincerely appreciate this valuable suggestion, which helps improve our work. The key advantage of LLMs lies in automatically discovering executable fusion primitives and fusion mechanisms beyond human designs. Some primitives involve unconventional designs that are difficult for human experts to think of. For example, one discovered primitive captures cross-view feature co-variation by modeling coordinated changes between neighboring feature dimensions. It is implemented as follows.

import tensorflow as tf
def primitive(inputs, fused_nb_feats):
    x = tf.stack(inputs, axis=0)
    mean = tf.reduce_mean(x, axis=0)
    centered = x - mean
    covariance = tf.reduce_mean(
        centered * tf.roll(centered, shift=1, axis=-1), axis=0
    )
    return mean + tf.tanh(covariance)

We compared the discovered library with 15 hand-designed operators under the same NAS budget. The former achieved 0.82 percentage points higher accuracy.

Operator libraryNTU RGB-D acc(%)
Expanded library of 15 hand-designed operators91.24
Operator library discovered by LLMDFO-NAS92.06

Q2 / W2. Statistical significance and search cost

Based on Table 1, the improvements over MBTF-NAS are relatively limited on some benchmarks. For example, the improvements on MM-IMDB and EgoGesture are only 0.27 and 0.08 percentage points, respectively. Are these improvements statistically significant? Besides, the search time increases from 0.41 to 2.33 hours on MM-IMDB and from 4.12 to 13.87 hours on NTU RGB-D. I wonder whether these improvements justify the additional search cost. A more detailed performance-efficiency analysis would strengthen the practical contribution.

Are the relatively small improvements on MM-IMDB and EgoGesture statistically significant across multiple independent search runs?

We thank the reviewer for these valuable concerns. We repeated our method three times. LLMDFO-NAS achieved 64.75 ± 0.04% weighted F1 on MM-IMDB and 95.36 ± 0.02% ACC on EgoGesture. The 95% confidence intervals were [64.65%, 64.85%] and [95.3%, 95.41%], respectively. All three runs outperformed the MBTF-NAS results (64.53% and 95.29%), indicating consistent performance. Although the gains are modest, they are comparable to previous methods. Specifically, our 0.08 gain exceeds the 0.04 gain of MBTF-NAS over CSG-NAS on EgoGesture. Our 0.27 gain is close to the 0.28 gain of CSG-NAS over CoMO-NAS on MM-IMDB. In addition, LLMDFO-NAS achieves higher weighted F1 with fewer trainable parameters on MM-IMDB.

Data and metricRun 1 (%)Run 2 (%)Run 3 (%)Average ± standard deviation(%)
MM-IMDB, weighted F164.8064.7364.7364.75 ± 0.04
EgoGesture, acc95.3895.3495.3595.36 ± 0.02
Data and metric95% confidence interval (%)MBTF-NAS acc (%)
MM-IMDB, weighted F164.65–64.8564.53
EgoGesture, acc95.30–95.4195.29

LLMDFO-NAS needs additional search costs due to its exploration of more diverse fusion primitives and the higher computational complexity of composed operators. However, the search is performed only once to identify the architecture. Moreover, LLMDFO-NAS uses 4 GPUs, and MBTF-NAS uses 8 GPUs. In terms of GPU-hours, LLMDFO-NAS's search costs are 2.84× and 1.55× those of MBTF-NAS on MM-IMDB and NTU RGB-D, respectively. These additional costs bring benefits. LLMDFO-NAS improves weighted F1 by 0.27 percentage points on MM-IMDB while reducing trainable parameters from 0.37M to 0.25M, and improves ACC by 0.81 percentage points on NTU RGB-D. This demonstrates a favorable trade-off between search cost and model performance.

AuthorsOpenReview ↗

Response to Reviewer hWFr (2/2)

Q3 / W3. Generalization of the zero-cost proxy

This paper uses an architecture-aggregated zero-cost proxy to evaluate the discovered fusion operators. However, the proxy mainly measures representational expressivity based on randomly initialized networks and Gaussian inputs. Does higher expressivity necessarily indicate better multimodal fusion? Although Figure 3 shows a strong correlation on EgoGesture, would this relationship remain consistent across other benchmarks and modality configurations? I think this is important for supporting the generalizability of the proposed evaluation strategy.

How does the architecture-aggregated zero-cost proxy correlate with downstream performance on benchmarks other than EgoGesture?

We appreciate the reviewer’s insightful comment. Higher expressivity does not necessarily guarantee better fusion, but it provides a useful signal for assessing the potential of the discovered fusion operators. We evaluated its rank correlation with downstream validation performance to examine the reliability of the proxy.

DatasetSpearman correlationKendall correlation
EgoGesture0.91030.7413
NTU RGB-D0.95000.7985
NUS-WIDE-1280.94500.7856

The consistently high correlations demonstrate strong agreement between proxy rankings and downstream performance rankings across different benchmarks. These results support the effectiveness of the proxy. It is a useful indicator rather than a guarantee of fusion performance.

Q4 / W4. Contributions of primitive discovery and composition search

Some experimental settings need further clarification. First, the proposed framework contains both operator discovery and architecture search, but the individual contribution of each stage is not sufficiently isolated. Would evolutionary composition search without LLM-generated primitives achieve similar results?

What is the individual contribution of LLM-based primitive discovery compared with evolutionary composition search?

We thank the reviewer for insightful comment, which motivates further analysis of the two stages. We evaluated all three methods on the same NTU RGB-D split under the same downstream NAS budget.

MethodNumber of operatorsAccuracy(%)
Expanded library of 15 hand-designed operators1591.24
Direct NAS with LLM primitives1591.67
Full LLMDFO-NAS592.06

LLM-based primitive discovery improves ACC from 91.24% to 91.67% (+0.43), demonstrating its contribution to providing effective fusion primitives for NAS. Further composition exploration by the LLM increases ACC to 92.06% (+0.39), highlighting the benefit of combining the discovered primitives into composite fusion operators. These results suggest that LLM-based primitive discovery provides additional benefits that evolutionary composition search alone may not achieve.

Q5 / W4. Analysis of operator usage

Table 5 shows that F5 accounts for 66.4% of the operator usage, while F4 contributes only 0.8%. Does this suggest that only a small subset of discovered operators is actually necessary? I think further analysis would help clarify the effectiveness of the learned operator library.

Since F5 dominates the discovered operator usage, would using only the top one or two operators achieve comparable performance?

F5 dominates operator selections (66.4%), followed by F3, while F1 and F2 are selected at comparable rates. F4 has the lowest selection rate. However, selection frequency does not fully reflect an operator's utility. The best NTU RGB-D architecture contains 14 fusion nodes, with six using F5, six using F1, and two using F2. It means that operators play complementary roles. To further evaluate the operator library, we compared different search spaces.

Operators availableNTU RGB-D acc(%)
F5 only91.44
F5 and F391.52
Full library without F491.82
Full five-operator library92.06

The full library improved accuracy by 0.54 percentage points over using only F5 and F3 (top two operators). Removing F4 also reduced accuracy by 0.24 percentage points, despite its low selection frequency. These results support the usefulness of retaining diverse fusion operators. Each gives NAS a different way to combine modality features, and even a rarely selected operator contributes to the performance achieved by the full library.

Thanks a lot for the authors' response. Most of my concerns have been resolved during the rebuttal. Considering the huge effort involved in this work, I would like to increase my rating, although I am still a little confused about the advantage of using LLMs for NAS.

Thank you very much for your positive feedback. The size and quality of the search space affect the potential performance of NAS. Our motivation is to leverage the generative capabilities of LLMs to create reusable fusion primitives and expand the search space. A key challenge is how to make effective use of LLMs to generate primitives. We further employ LLM-guided evolutionary search to explore primitive compositions. Notably, our experiments demonstrate that LLMDFO-NAS can discover effective primitive combinations beyond those manual designs. We will further clarify this motivation in the manuscript. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.