- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes LLMDFO-NAS, a framework for multimodal classification that integrates fusion operator discovery with neural architecture search (NAS). Unlike traditional multimodal NAS methods, which search for network architectures within a fixed library of fusion operators, this approach first leverages a large language model (LLM) to generate fusion primitives. These primitives undergo rigorous validation to ensure their viability in terms of execution, shape compatibility, gradient propagation, and determinism. Subsequently, these primitives are combined into complex fusion operators via evolutionary search; during this process, a "zero-cost" proxy metric, based on the expressive capacity of an operator within various architectural contexts, is employed alongside a Pareto selection mechanism to filter the generated operators. The resulting library of operators is then utilized in the subsequent search for multimodal fusion architectures, effectively expanding the search space to encompass both fusion functionality and network architecture. The proposed method is evaluated against various existing multimodal NAS approaches across four major benchmarks. Additionally, supplementary experiments are conducted on the MV-RML2016.10A dataset, focusing on the classification of wireless signals under low signal-to-noise ratio (SNR) conditions.
Critical strengths
-
Expanding the scope of the search space offers notable methodological value. Traditional multimodal Neural Architecture Search (NAS) typically treats fusion operators as predefined functional units, focusing primarily on the connections, interaction sequences, and overall architectural structure. In contrast, this paper represents the fusion operators themselves as executable programs, employing a combination of Large Language Models (LLMs) and evolutionary search to automatically construct these operator combinations.
-
The experiments encompass a diverse range of multimodal configurations, with preliminary validation provided for proxy metrics. The paper’s four primary benchmarks cover various input configurations, including image-text, multi-view visual data with labels, RGB-depth, and video-skeleton data. Furthermore, the authors report a strong correlation on the EgoGesture dataset between the rankings derived from zero-cost proxies and the validation performance achieved after full training. This evidence suggests that the authors have not only proposed a framework for searching fusion operators but have also sought to verify the predictive validity of their core selection mechanism.
Critical weaknesses
-
Certain aspects of the problem formulation and experimental protocols are insufficient to fully substantiate the validity of the results. 1.1: There is an inconsistency between the multi-label task definition for the MM-IMDB dataset and the format of the model's outputs that remains to be addressed. 1.2: The normalization axes for the operators described in Appendix F5 may not align with the functions claimed in the paper.
-
While the authors conduct ablation studies on the LLM backbone and the evolutionary composition strategy, and report on the correlation between zero-cost proxies and training performance, the current evidence fails to definitively answer the following key questions: Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition? Furthermore, given a fixed budget for evaluating candidate solutions, do zero-cost proxies perform better than random selection?
Questions for authors
-
Certain aspects of the problem formulation and experimental protocols are insufficient to fully substantiate the validity of the results. 1.1: There is an inconsistency between the multi-label task definition for the MM-IMDB dataset and the format of the model's outputs that remains to be addressed. 1.2: The normalization axes for the operators described in Appendix F5 may not align with the functions claimed in the paper.
-
While the authors conduct ablation studies on the LLM backbone and the evolutionary composition strategy, and report on the correlation between zero-cost proxies and training performance, the current evidence fails to definitively answer the following key questions: Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition? Furthermore, given a fixed budget for evaluating candidate solutions, do zero-cost proxies perform better than random selection?
-
The paper constructs zero-cost proxies using random Gaussian inputs and covariance spectral entropy, and incorporates primitive complexity into the Pareto selection process. Please provide a precise numerical definition for , explain how these values are assigned, and clarify whether they are generated by the LLM itself.
3 comments
Thank you very much for your rebuttal. However, my concerns remain unaddressed, so I will keep my score unchanged.
We sincerely apologize that our previous response may not have fully addressed your concerns. We greatly appreciate your constructive feedback. To better address your concerns and provide stronger supporting evidence, we have conducted additional experiments and further analysis.
Q2.1. Given the same library of primitives, does LLM-based evolutionary composition outperform random composition or conventional evolutionary composition?
We thank the reviewer for this insightful comment, which motivate us to further analyse the role of LLMs in our method. We compare three strategies using the same library of 15 LLM-generated primitives on NUS-WIDE-128. The results are reported below.
| Method | NUS-WIDE-128 ACC(%) |
|---|---|
| Random composition | 76.08% |
| Conventional evolutionary composition | 76.19% |
| LLMDFO-NAS | 76.69% |
Notably, conventional evolutionary composition achieves only a small improvement of 0.11 percentage points over random composition. It means that conventional evolutionary operations provide limited benefits in exploring primitive combinations. In contrast, LLMDFO-NAS achieves improvements of 0.61 and 0.50 percentage points over random and conventional evolutionary composition, respectively. It means that LLM guidance provides more effective direction for exploring the composition space. These results further support the effectiveness of LLM-based evolutionary composition.
About Q2.2: We sincerely apologize for the incorrect data in our previous response. This error was caused by an incorrect parameter setting in the code. We noticed this issue during a subsequent check after submitting our response. However, we received your feedback while we were in the process of correcting the data. The corrected results have now been updated in our previous response. We sincerely regret this oversight and apologize for any confusion and inconvenience caused, as well as for taking up your valuable time.
Q3. The paper constructs zero-cost proxies using random Gaussian inputs and covariance spectral entropy, and incorporates primitive complexity (c(o)) into the Pareto selection process. Please provide a precise numerical definition for (c(o)), explain how these values are assigned, and clarify whether they are generated by the LLM itself.
During primitive generation, the LLM is provided with the fusion task, the allowed tensor operations, and the implementation constraints. It generates the executable code for each primitive and assigns a complexity score (c(o)) based on its own judgment. The prompt specifies the scoring range [0, 4] and leaves the scoring criteria to the LLM. Our code checks the range and stores the returned score as a fixed cost for that primitive. During evolutionary search, the complexity of a composition is calculated as the sum of the costs of its constituent primitives.
Finally, we would like to further clarify our motivation. The size and quality of the search space affect the potential performance of NAS. Our motivation is to leverage the generative capabilities of LLMs to create reusable fusion primitives and expand the search space. A key challenge is how to make effective use of LLMs to generate primitives. We further employ LLM-guided evolutionary search to explore primitive compositions. Notably, our experiments demonstrate that LLMDFO-NAS can discover effective primitive combinations beyond those manual designs. We will further clarify this motivation in the manuscript. We hope these revisions adequately address your concerns. Please feel free to let us know if you have any further questions or concerns. We sincerely thank you again for your time and valuable suggestions.
Response to Reviewer LfF8
We thank the reviewer for the constructive feedback, which will help improve the quality of our manuscript.
Q1. Multi-label prediction on MM-IMDB and normalization axes in F5.
We sincerely apologize for these oversights and will clarify the relevant parts in the revised manuscript. About 1.1: The softmax shown applies to single-label tasks. For MM-IMDB, we instead use 23 independent logits, and trained with weighted binary cross-entropy. So each genre is predicted independently. We used these predictions to compute the weighted F1 reported in Table 1. About 1.2: The axis=0 takes the maximum across views in the first operation. Once the view dimension is removed, axis=0 refers to the batch dimension. The second operation uses the current batch to compute a mean and variance for each feature and standardizes each feature separately. It means that our method perform maximum aggregation across views, followed by normalization of each feature across the batch. We will clarify them in the revised manuscript.
Q2. Guided Fusion Operator Discovery vs. Random Composition
We thank the reviewer for insightful comments, which motivate us to further analyse the role of LLMs in our method. Both methods used the same library of 15 LLM-generated primitives. For the random baseline, we constructed five fusion operators by randomly selecting and connecting two to four primitive nodes per operator. We evaluated both five-operator libraries using the same downstream NAS procedure on NUS-WIDE-128. The result shows that LLM-guided composition produced a more effective fusion library from the same primitives in this experiment.
To evaluate the effectiveness of the zero-cost proxy, we ranked 200 held-out candidate compositions according to their zero-cost proxy scores and validation accuracy after training on the other three datasets. The results are reported below.
The two rankings agreed closely across all three benchmarks. We further compared proxy selection with random selection under different evaluation budgets.
For each budget, we selected the top-k candidates ranked by proxy scores and calculated their mean accuracy. Random selection uniformly sampled k candidates without replacement from the same pool and averaged their accuracies. This process was repeated 1000 times, with results reported as mean ± SD. Proxy selection consistently achieved higher mean accuracy than random selection across all three datasets, especially on NUS-WIDE. It demonstrates the effectiveness of the zero-cost proxy in identifying promising candidate compositions.
Q3. Primitive complexity c(o)
During primitive generation, the LLM sees the fusion task, the allowed tensor operations, and the implementation constraints. It generates the code for each primitive and uses its own judgment to estimate how complex the resulting computation is. The prompt sets the score range and leaves the scoring criteria to the LLM. Our code checks the range and stores the returned score as a fixed cost for that primitive.