HeteroWeave: Variable-Cardinality Block Composition in Pretrained Model Reassembly

Existing heterogeneous pretrained model reassembly methods usually retain one block at each aligned position, limiting the potential for different pretrained sources to contribute jointly. Allowing multiple blocks to participate at the same position increases reassembly flexibility, but turns block identity and participation cardinality into coupled architectural decisions, substantially expanding the composition space. Meanwhile, additional parameter count and computation do not translate monotonically into performance gains, creating explicit tradeoffs between performance and resource cost. To address these challenges, we propose HeteroWeave to reformulate heterogeneous pretrained model reuse as a variable-cardinality, multi-objective reassembly problem that explicitly accounts for performance–resource trade-offs. It jointly determines block identity and participation cardinality within a unified composition space, while shared interfaces and lightweight adaptation make heterogeneous compositions executable. To avoid training every candidate, we design a low-cost proxy that estimates performance from block-level activation-pattern statistics, using square-root aggregation to reduce score inflation as the activation space expands. HeteroWeave then optimizes the proxy, parameter count, and FLOPs as three objectives via Pareto-guided evolutionary search, retaining diverse nondominated architectures for different training and inference preferences. We validate HeteroWeave across four tasks, covering image classification, vision-language retrieval, semantic segmentation, and object detection. On classification, one nondominated representative improves accuracy by 1.08 percentage points over the single-block reassembly baseline, while using 26.7% fewer parameters and 11.9% fewer FLOPs.