- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes a model embedding memory for robot policy learning. It uses the SVD of pretrained weights to build a shared embedding space, where task-specific adapters can be stored and compared. A compact index is used for efficient retrieval, and language instructions can also be mapped into this space to retrieve relevant policies. Experiments on Meta-World, LIBERO, and RoboCasa show promising results in parameter efficiency, retrieval, and task performance.
Critical strengths
-
The idea of using task-specific adapters as a searchable memory is interesting and practical. It connects parameter-efficient fine-tuning with policy retrieval in a simple framework.
-
The experimental results are promising. The method achieves comparable performance to LoRA with fewer parameters, and the retrieval experiments show clear efficiency gains.
Critical weaknesses
-
The novelty seems somewhat limited. SVD-based fine-tuning has already been explored in works like SVDiff and Spectral Adapter, while model and skill retrieval have also been studied. The main contribution seems to be combining these ideas, but the paper does not clearly show why this particular combination is better than existing alternatives.
-
Some important baselines and ablations are missing. For example, using a LoRA adapter bank with the same retrieval mechanism would be a natural baseline. Also, it is unclear whether the pretrained SVD basis is actually necessary, since a shared random orthogonal basis might provide similar geometric properties. The current experiments do not fully isolate where the improvements come from.
Questions for authors
-
How does the method compare with Spectral Adapter or a LoRA adapter bank using the same retrieval mechanism and task-specific training setup?
-
Have you tried using a random fixed orthogonal basis instead of the pretrained SVD basis? Would the task clustering and retrieval performance still hold?
4 comments
Rebuttal (Part2)
(W2, Q2) Compare with the shared random orthogonal basis.
A: To investigate whether pretrained spectral directions are necessary, we replace the SVD bases with fixed random orthogonal bases while preserving the block structure and trainable parameter budget.
| Method | Spatial | Object | Goal | LIBERO-10 | Overall | Params |
|---|---|---|---|---|---|---|
| Model Embedding (Random basis) | 79.0% | 56.0% | 68.0% | 23.0% | 56.5% | 0.9M |
| Model Embedding (SVD basis) | 91.5% | 88.5% | 90.5% | 74.0% | 86.1% | 0.9M |
Replacing the pretrained SVD basis with a random orthogonal basis reduces overall success from 86.1% to 56.5%, with consistent degradation across all four LIBERO suites. Since both variants use identical block structures and trainable parameter counts, this result isolates the contribution of the basis choice rather than model capacity.
The results indicate that the pretrained SVD basis provides task-effective adaptation directions that cannot be replaced by an arbitrary shared orthogonal basis without substantial performance degradation under the evaluated configuration.
To further assess the impact of pretrained versus random orthogonal bases on retrieval performance, we compare language-guided retrieval accuracy under both settings.
| Method | Top-1 | Top-3 | Top-5 |
|---|---|---|---|
| Ours (random basis) | 92.5% | 97.0% | 98.0% |
| Ours (SVD basis) | 94.0% | 99.50% | 99.50% |
The pretrained SVD basis consistently outperforms the random basis, improving Top-1, Top-3, and Top-5 retrieval accuracy. Notably, the random basis still achieves strong retrieval accuracy, suggesting that a shared coordinate system supports task discrimination even without pretrained spectral alignment. However, the pretrained basis provides additional benefits for retrieval, indicating that its structure contributes to more effective task representations.
Combined with the adaptation results, these findings demonstrate that pretrained spectral structure benefits both task adaptation and retrieval, with a substantially larger impact on adaptation performance.
Additional experimental results: In our response to reviewer wAsr, we also include the other experiments.
-
Spectral-based methods (wAsr, Part1): Additionally, we also compare our method with Spectral Rewiring.
-
Retrieval experiments (wAsr, Part3): We compare our language-guided retrieval method with baselines including text-based nearest-neighbor retrieval, a supervised task classifier, LoraRetriever, RAMoLE, Arrow-style routing, and a reusable LoRA adapter library (MBC).
Thank you for the detailed response and additional experiments. The comparisons with Spectral Adapter, SVDiff, and LoRA under the same retrieval setting address my main concerns. In particular, the random orthogonal basis experiment provides convincing evidence that the pretrained spectral structure is important for effective task adaptation, rather than simply providing a shared coordinate system.
While I still think the methodological novelty is somewhat incremental, the new results better establish the benefits of the proposed representation in terms of adaptation performance and parameter efficiency. My main concerns have been largely addressed, and I am raising my rating to weak acceptance.
Thank you for your positive feedback and support. We sincerely appreciate your time and thoughtful evaluation of our work.
Rebuttal (Part1)
We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.
(W1, Q1) What is the contribution of this paper compared to existing work? Compare with the spectral-adaptation methods and other retrieval methods.
A: Our contribution is investigating how to construct a structured model embedding space in which task-specific adaptations can be learned, represented (Sec. 2.1), organized, and retrieved (Sec. 2.2-2.4, Sec. 3) within a shared coordinate system derived from the pretrained model.
Comparison with Spectral-adaptation methods. Let the pretrained weight matrix have SVD W0=UΣV⊤. Model embedding directly optimizes task-specific coordinates Mi(h) within the subspace, spanned by the pretrained bases U(h) and V(h): ΔWi=∑U(h)Mi(h)V(h)⊤.
Spectral Adapter introduces additive and rotational adaptations of selected singular vectors. Its formulation is Wi=(U+ΔUi)Σ(V+ΔVi)⊤, where only selected singular-vector columns are modified. In contrast, model embedding keeps U and V fixed.
SVDiff adapts pretrained weights by updating only their singular values while keeping the singular-vector bases fixed Wi=Udiag(ReLU(σ+δi))V⊤. In contrast, Model Embedding learns Mi within structured pretrained subspaces, enabling interactions between spectral directions and providing greater adaptation expressivity than SVDiff's diagonal-only parameterization.
We further evaluated Spectral Adapter and SVDiff on LIBERO to investigate the differences between these formulations. All results are obtained using oracle task-ID selection, without retrieval, to isolate task-specific adaptation performance.
At a matched parameter budget of 0.9M, our method improves average success over Spectral Adapter. SVDiff provides a much smaller adaptation budget but achieves substantially lower success, indicating that its restricted adaptation capacity is insufficient for these robotic policy-learning tasks under the evaluated configuration.
Same retrieval mechanism for other adaptation methods. To further isolate the contribution of the proposed representation from task-specific adaptation and retrieval, we conducted additional controlled comparisons against Spectral Adapter, LoRA, and full fine-tuning on LIBERO, using the same MTACT backbone and experimental protocol described in Sec. 4.2.
We consider two settings: (i) oracle task-ID selection, where the correct task-specific adapter is directly selected; and (ii) using the same language-guided retrieval mechanism to select the adaptation.
Oracle task-ID selection:
Language-guided retrieval mechanism:
The performance gap between oracle task-ID selection and actual retrieval is small, as the language-guided router achieves nearly perfect task identification. Only one task in LIBERO-10 is assigned an incorrect adapter, leading to a slight performance degradation.
Model Embedding demonstrates effective task adaptation with much fewer parameters. Under oracle selection, it achieves 86.1% overall success, outperforming LoRA and Spectral Adapter. It also approaches full fine-tuning performance with significantly fewer parameters. These results isolate the contribution of the adaptation representation. The performance advantage persists under actual retrieval. With the shared retrieval mechanism, Model Embedding achieves 85.4% overall success compared to the LoRA adapter bank, while retaining a smaller per-task parameter budget.
These results demonstrate that our structured model embedding space provides a favorable trade-off between adaptation performance and parameter efficiency.