- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HeteroWeave constructs models by selecting one or more pretrained blocks at each aligned position. Adapters reconcile incompatible interfaces, and selected block outputs are fused. A training-free activation score, CLAS, ranks candidate architectures, while a multi-objective evolutionary search balances this score against parameter count and FLOPs. The paper evaluates ImageNet classification and additional compositions for retrieval, segmentation, and detection.
Critical strengths
-
The main ImageNet result improves accuracy and resource use together. The medium-size model has good gains over DeRy while reducing parameters and FLOPs significantly.
-
The capacity-matched experiment directly examines whether parallel composition helps beyond adding parameters. The parallel extension outperforms the serial extension under nearly identical resource costs. The cross-task results also show that adding more blocks is not uniformly beneficial, which supports treating participation count as a search variable.
Critical weaknesses
-
CLAS is validated against accuracy after only one or five training epochs. However, the paper itself shows that early performance can differ from the final ranking. The partial correlations and seed-stability checks are useful, but they do not establish that CLAS reliably identifies the best architectures after the full 100-epoch training schedule.
-
The contribution of variable cardinality is not fully separated from the rest of the search pipeline. Comparing against DeRy changes more than the allowed number of blocks, while the capacity-matched experiment tests particular extensions rather than the searched architecture space. A single-block version using the same CLAS score, adapters, search budget, and selection rule would provide a cleaner test.
Questions for authors
Following form the two weaknesses:
-
How well does CLAS correlate with fully trained accuracy on a representative candidate subset? Does square-root aggregation improve this correlation relative to the same score without that transformation?
-
What happens when the complete search is repeated with cardinality restricted to one at every position, keeping the source pool, resource limits, evaluation budget, and selection rule fixed?
3 comments
Response to Rebuttal
Thanks for doing these tests. The results after 100 epochs answer my question about CLAS and the square root. The single-block test also shows that allowing more blocks helps. I have increased my score to 4.
Dear Reviewer 5Lor,
Thank you for considering our rebuttal and updating your score. We are glad that the additional experiments addressed your concerns. Your constructive feedback has helped us strengthen the evaluation.
Thank you again for your time.
Response to W1–W2 and Q1–Q2: CLAS Ranking and Variable Cardinality
W1 / Q1 — CLAS ranking and square-root ablation
Thank you for these thoughtful questions. We evaluate all proxies against 100-epoch ImageNet-1K results of 20 reassembled architectures, including CLAS without the square root. All proxies use the models’ initial states.
Table C.1: Spearman correlation with fully trained Top-1 accuracy. Values are means ± sample SD over three proxy-scoring seeds.
(a) Final-performance ranking. CLAS achieves the highest mean Spearman correlation, supporting its ranking utility beyond short training.
(b) Square-root benefit. Removing only the square root reduces correlation from 0.81 ± 0.02 to 0.71 ± 0.01. The transformation assigns diminishing credit to additional patterns within each module, tempering count-driven score growth.
Revision. We will add these results to Section 4.2.2 (Figure 3a) and Appendix B.1/B.3 (Table 7), and clarify aggregation in Section 3.3.2.
W2 / Q2 — Contribution of variable cardinality
We thank the reviewer for this suggestion. We add a complete-search single-block baseline to assess the contribution of variable cardinality.
We fix block cardinality to one at every position while retaining block-identity search. The source pool, CLAS score, adapters, 30M-parameter/6G-FLOP limits, total search budget, and selection rule remain identical. Both searches use NSGA-III with a population of 96 for 160 generations. We select the highest-CLAS composition from each feasible Pareto front, then fully fine-tune both selected architectures on ImageNet-1K for 100 epochs using the same recipe and training seed 0.
Table C.2: Complete-search single-block comparison under the 30M-parameter/6G-FLOP limits.
Variable cardinality improves the searched architecture. Under matched search and training settings, it achieves 1.05 percentage points higher Top-1, with 30.3% fewer FLOPs and 1.5% more parameters.
Revision. We will add this comparison and its protocol to Section 4.2.1 (Table 4).
Thank you again for your thoughtful suggestions. The additional comparisons have made our evaluation more complete.