- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Representax, an open-source JAX framework for training representation learning in text, vision and multimodalities. By abstracting the representation learning problem with four layers, namely model& processors, learning tasks, data distributions, and evaluations, it transforms a representation learning problem into a job configuration across the four layers, making it easy to implement different learning tasks across different modalities within supported configurations. The framework covers dense and late-interaction retrieval, sentence-pair learning, cross-encoders, reward modeling, multimodal contrastive learning, and V-JEPA-style predictive learning. The authors evaluated Representax in three parts:
- Comparisons with established implementations: thirtheen recipes against PyTorch-based references on one RTX 4090 and on a 16-chip TPU v5e slice.
- end-to-end adaption studies: dense retrieval, CLIP on CoCo and then Flicker30k, and three-recipe multimodal adaption of Jina v5 Omini Nano.
- Fixed-work multi-accelerator scaling on 1-8 A100 GPUs.
Overall, the paper is well written and easy to follow.
Critical strengths
- This paper provides a unified framework for text and multimodal representation learning, across different tasks, such as dense retrieval, similarity ranking, reward modeling, multimodal contrastive learning, masked language modeling, and joint-embedding prediction. Under the framework, it is easy to experimenting different learning methods for different tasks. Reducing the workload required when switching learning objectives.
- Validation experiments are solid. It compares thirteen paired workload configurations on GPUs and TPUs, end-to-end adaptation studies for dense retrieval and CLIP, and fixed-work strong scaling.
Critical weaknesses
- While the abstraction layers that Representax presented are well thought, it did not provide analysis or experiments on how different design can affect the choices of components and how specific design choices are made, what are different design choices to make, etc.
- The scientific contribution of this paper limited for the ICLR main track. It does not show new research enabled by this framework. A more framework/system focused venue (i.e. MLSys) could be a better fit.
Questions for authors
- Could you provide an example of changing new objective or model extension, and show specific changes to be made to reuse the overall training, evaluation, and execution lifecycle?
- Could you compare the framework to established JAX baseline for dense retrieval, CLIP or other tasks?
Thank you for these questions!
Section 3.5 now shows how we added a trainable projection to MPNet. We wrote a model wrapper and a loader, then selected the embedding size and training objective in the config. Across all 18 runs, we reused the same training, evaluation, and checkpoint code. We also checked that training resumed from a checkpoint and that each exported model reproduced its outputs on the test inputs. Appendix I.3 includes the full code we added.
We added a comparison against Hugging Face’s Flax CLIP in Appendix I.1 using the same batches, objective, and optimizer. Flax was faster in all five FP32 runs. BF16 was closer: Flax had higher median throughput (812 vs. 783 examples/s), while Representax was faster in two of the five runs. These measurements cover the training step and exclude preprocessing, evaluation, and checkpointing.