acceptodds
Withdrawn from ICLR 2027

Reconfigurable Embodied Reasoning: Offline Composition of Replaceable Capability Modules

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generalist embodied models typically integrate capabilities such as grounding and interaction, spatial reasoning, temporal understanding and planning, and video understanding into a single checkpoint. We explore how these capabilities can be independently maintained, updated, and recombined after training while retaining deployment as a single model. We organize independently maintained LoRA modules into a capability library and introduce ReCAP, a weight-only composition method that integrates selected modules into one static checkpoint. ReCAP incorporates capability updates through module replacement and recomposition, without retraining the remaining modules. ReCAP anchors composition in one complete module update and expresses the others as residual edits in shared coordinates. It aggregates these edits through magnitude-weighted voting in a joint residual subspace and controls their injection with a support mask. ReCAP achieves the highest Full-11 score among the evaluated training-free merging methods across 11 embodied reasoning benchmarks, two backbones, and three capability-library settings. On libraries trained from a shared base, it scores 66.53 on InternVL3.5-14B and 68.27 on Qwen3-VL-8B, exceeding the strongest single modules by 1.86 and 0.85 points. Under a shared evaluation pipeline, the Qwen3-VL-8B composition outperforms all nine public embodied models, leading the strongest comparator by 2.89 points. Further experiments show that task-specific recomposition improves target performance, local module improvements benefit the composed model without retuning, and incremental updates obtained through continued training of an individual capability can enter an existing capability library. These results demonstrate the viability of building high-performing embodied models from independently evolving capability modules while preserving single-checkpoint deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied Reasoning, Vision-Language Models, LoRA Composition, Training-Free Model Merging, Modular Capability Updates

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JtXPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to merge different specialist LoRA modules into a unified model. Specifically, it selects a reference module and donor modules, and merges the donors into the reference using a joint residual subspace, magnitude-weighted voting, and support-guided injection. Table 1 presents the primary results supporting this merging algorithm, showing slightly better performance than several merging baselines.

Critical strengths

Merging different capability modules into a single checkpoint for embodied reasoning tasks is a promising direction.

The experiments cover different baselines and settings, including different choices of reference modules in Table 2, and are generally comprehensive.

Critical weaknesses

The introduction does not clearly explain the specific challenges addressed by this method or its practical value beyond existing model-merging approaches.

The differences between the proposed method and the baselines are unclear, as several components substantially overlap with Task Arithmetic, TIES, KnOTS, and Core Space. In addition, Twin-Merging appears to be related but is neither cited nor discussed. The author should describe the difference and extension more clearly.

The experiments do not sufficiently support claims about broader embodied capabilities, as they mainly evaluate spatial reasoning, localization, and video understanding. Merging modules for tactile sensing, robot action learning, and additional perception capabilities would be more important for demonstrating broader embodied capabilities.

The performance gains over the merging baselines, specialist models, and joint training appear relatively small. It would be helpful to explain the practical benefits of this work in more detail.

It would be better to discuss the limitations and possible extensions more explicitly in a dedicated paragraph in the main text.

Questions for authors

Please see the weaknesses above for details.

I am curious whether this approach could be applied to improve multi-task training by making updates more consistent, and whether it could also benefit model pruning.

Is there a theoretical upper bound on the performance of this method?

Reviewer_X7hEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper organizes multiple embodied capabilities of a VLM (grounded interaction, spatial reasoning, temporal planning, and video reasoning) as independently trained LoRA modules that form a replaceable capability library, and proposes ReCAP, a training-free method that composes selected modules offline into a single static model. Working in the common coordinates of Core Space, ReCAP takes one module as the reference, expresses the remaining modules as residuals, aggregates these residuals by magnitude-weighted sign voting within a joint subspace, and controls their injection with a consensus mask. Experiments on InternVL3.5-14B and Qwen3-VL-8B across 11 embodied benchmarks show that ReCAP's Full-11 score exceeds the strongest existing merging method in each setting by 0.56–0.74 points, and also surpasses the strongest single module and joint training. The authors further use task-specific recomposition, module replacement, and continued-training increment experiments to argue that the capability library can be reused and evolved.

Critical strengths
  1. Broad and well-documented evaluation. Experiments cover two backbones, 11 embodied benchmarks, and three capability-library settings, with comparisons against four training-free merging methods, single modules, a per-benchmark oracle, and joint training. ReCAP ranks first among the evaluated training-free methods in every setting, and the appendix reports all composition configurations and per-benchmark results.
  2. Practically relevant library-evolution questions. Beyond standard multi-module merging, the paper systematically examines module replacement, the entry of increments obtained by continued training from an adapted model, and coefficient handling after a full-library refresh. These scenarios arise in real deployments but are rarely examined in the merging literature, and the paper offers some useful observations, e.g., that reusing the original coefficients works better after a single-module replacement, whereas norm calibration is needed after a full refresh.
Critical weaknesses
  1. Limited methodological novelty. ReCAP builds on the existing Core-Space TIES pipeline (Sec. 3.1, 3.4), and several of its core components have clear precedents:

    • The idea of computing residuals relative to a chosen model, processing them in a low-dimensional subspace, and injecting them back (Sec. 3.2–3.3) appears in CART [1] (anchored at the weight average) and Twin-Merging [2] (anchored at a merged shared model). However, the paper neither cites nor compares against either work.
    • The objective of the joint residual subspace (Sec. 3.3, Eq. 4) and its alternating eigendecomposition solver (App. B) are essentially identical to GLRAM [3]. The paper does not cite this work, yet re-derives it over several pages in App. B; I suggest condensing this appendix and citing GLRAM instead.

    Overall, the method is largely a combination and application of existing techniques, with its novelty concentrated in specific design choices: anchoring on a single expert, learning a shared subspace across all donor residuals, and consensus-mask-based injection (Sec. 3.5). These choices are largely heuristic, and their advantages over natural alternatives have not been sufficiently established.

  2. Unclear hyperparameter selection and fairness of comparison. ReCAP involves numerous hyperparameters, and several of them (donor coefficients, the choice of reference, and ηoff\eta_{\text{off}}) take different values on the two backbones. However, the paper does not explain how they were selected, provides no sensitivity analysis, and does not report the hyperparameters of the baselines. The paper’s own results show that performance is highly sensitive to the coefficients: after a full-library refresh, adjusting the donor coefficients alone changes Full-11 by 2.02 points, and mismatched coefficients reduce it by 6.20 points, both far exceeding ReCAP’s margin over the baselines (0.56–0.74). It is therefore unclear whether this margin stems from the method itself or from more extensive tuning.

  3. Although ReCAP outperforms all compared merging methods on Full-11 in every setting, this advantage comes mainly from small-scale benchmarks. On the five larger benchmarks (each with more than 1,600 items), ReCAP’s average margin over the strongest baseline is only +0.02, +0.26, and +0.12 in the three settings, with wins and losses on both sides. In contrast, the six small-scale benchmarks account for over 80% of the summed per-benchmark differences. For example, on the Qwen3-VL-8B backbone (ReCAP vs. KnOTS-TIES), the three smallest benchmarks alone (100–300 items each) contribute about 88% of the summed difference.

References:

[1] Jiho Choi, Donggyun Kim, Chanhyuk Lee, and Seunghoon Hong. Revisiting weight averaging for model merging. arXiv preprint arXiv:2412.12153, 2024.

[2] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-Merging: Dynamic integration of modular expertise in model merging. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

[3] Jieping Ye. Generalized low rank approximations of matrices. Machine Learning, 61:167–191, 2005. (An earlier version appeared in ICML 2004.)

Questions for authors
  1. ReCAP’s methodological novelty lies mainly in three design choices: anchoring on the full update of a single expert, learning a shared subspace across all donor residuals, and consensus-mask-based injection. However, their advantages over natural alternatives have not yet been tested. Keeping the rest of the pipeline unchanged, could the authors add the following controls to demonstrate the advantages of these designs?

    • Replacing single-expert anchoring with anchoring at the weight average;

    • Replacing the shared residual subspace with separate truncation of each residual;

    • On the primary InternVL3.5-14B setting, replacing the consensus mask with a random mask of the same density and with uniform-scaling injection (Ma+cT⋆augM_a+cT_\star^{\mathrm{aug}}) .

  2. How were ReCAP’s hyperparameters (e.g., donor coefficients, the choice of reference, and ηoff\eta_{\text{off}}) selected, and was a validation split independent of the evaluation benchmarks used? How were the hyperparameters of the baselines set and tuned? In addition, could the authors report ReCAP’s sensitivity to the donor coefficients and ηoff\eta_{\text{off}}, to show that its margin over the baselines does not stem from more extensive tuning?

  3. Could the authors report paired bootstrap confidence intervals or significance tests for the difference between ReCAP and the strongest baseline, based on the existing per-item predictions? For benchmarks evaluated with sampled decoding (e.g., ERQA), could they also report the mean and variance over multiple runs?

Reviewer_JYhBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the composition of multiple capability modules for embodied reasoning and proposes ReCAP, a weight-only composition method. ReCAP combines pretrained LoRA modules in a shared low-dimensional space to integrate embodied reasoning capabilities and support module replacement. It selects one module as a reference and constructs reference-relative residuals from the remaining modules. These residuals are combined through subspace projection, sign-consistent merging, and residual compensation. The authors demonstrate the method’s effectiveness on 11 benchmarks across two model scales.

Critical strengths

The paper addresses a problem of practical significance. The empirical evaluation is solid, covering 11 benchmarks across two model scales. The paper includes a joint-training baseline matched in total LoRA rank and training token budget, providing a well-controlled comparison.

Critical weaknesses

I have four concerns about this paper, with the first three constituting my main concerns.

The discussion of related work is incomplete, and the technical contribution needs to be clarified. I think the contribution of this paper appears incremental. [1, 2] have already studied bilateral subspace optimization. In particular, maximizing the sum of projected energies and alternating between updates of the left and right orthonormal bases are closely related to them. The authors should clearly distinguish standard matrix approximation techniques adopted in this work from new designs specifically developed for capability-module composition. The difference with [3] should also be discussed. The proof section reads strongly like AI-generated text that has not undergone sufficient human editing. Although the authors disclose AI assistance, including assistance with checking mathematical expressions and proofs, my impression remains that this section reads like a first draft written entirely by AI, without adequate human revision. Numerous steps are individually numbered, many equation numbers are never subsequently referenced, and boxed expressions repeatedly highlight intermediate results that could be stated directly. This presentation makes the proofs unnecessarily lengthy and obscures the substantive theoretical content. The authors should substantially condense the standard derivations and clarify their actual significance for the proposed method. The limit of five rounds of alternating updates lacks sufficient justification. The authors discuss the monotonicity of the objective under alternating updates, but this does not establish that five rounds are sufficient or that the resulting solution is robust to initialization and changes in the module set. Performance is sensitive to reference selection, but the practical selection criterion is unclear. The paper includes some experiments with different reference modules, but these results also reveal substantial sensitivity to this choice. In Table 15, changing the reference within the same module set changes Mean9 from 65.22 to 66.79, a difference of 1.57 percentage points. This exceeds the improvements over the strongest baselines reported in the main experiments. The rationale for reference selection and its implications for practical use require further explanation. References

[1] Generalized low rank approximations of matrices. ICML, 2004.

[2] Dimensional Singular Value Decomposition (2DSVD) for 2D Maps and Images. SDM, 2005.

[3] DC-Merge: Improving Model Merging with Directional Consistency. CVPR, 2026.

Questions for authors

What are the differences between this method and prior work? Compared with existing subspace optimization and model merging methods, what unique challenges does capability-module composition introduce, and how do these challenges motivate the new techniques? What guarantees do the theoretical results provide for the complete algorithm? The current analysis primarily addresses projection optimality and alternating updates, whereas the full method also includes sign selection, trimming, masking, and residual compensation. Which theoretical guarantees remain hold after these operations are introduced? How should the reference module be selected in practice? Given the substantial performance variation across reference choices, how were the default references in the paper determined? Are other capabilities preserved after a module upgrade?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.