acceptodds
Under review as a conference paper at ICLR 2027

Representax: Scalable Representation Learning in JAX

Abstract

Learned representations shape how models organize information and generalize beyond their training examples. As learning systems evolve to incorporate information from increasingly diverse modalities, a flexible and efficient foundation for representation-learning research becomes essential to enable progress. We introduce Representax, an open-source JAX framework for developing and training representation-learning models across modalities. Representax unifies extensible training and evaluation tasks with established methods spanning dense and late-interaction retrieval, multimodal contrastive learning, reward modeling, and joint-embedding predictive learning. Its shared training infrastructure supports model adaptation, multimodal data processing, memory-efficient optimization, and distributed execution on GPUs and TPUs. To demonstrate its capabilities, we compare training performance against established frameworks, train and evaluate retrieval and multimodal models, and measure multi-GPU strong scaling. These experiments demonstrate competitive throughput, improvements on held-out retrieval tasks, and efficient scaling. Representax provides a foundation for investigating how representations are learned and adapted, and for taking new ideas from exploration to scale.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
representation learning, multimodal learning, JAX, contrastive learning, distributed training, research software

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GJ4pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Representax, an open-source JAX framework for training representation learning in text, vision and multimodalities. By abstracting the representation learning problem with four layers, namely model& processors, learning tasks, data distributions, and evaluations, it transforms a representation learning problem into a job configuration across the four layers, making it easy to implement different learning tasks across different modalities within supported configurations. The framework covers dense and late-interaction retrieval, sentence-pair learning, cross-encoders, reward modeling, multimodal contrastive learning, and V-JEPA-style predictive learning. The authors evaluated Representax in three parts:

  1. Comparisons with established implementations: thirtheen recipes against PyTorch-based references on one RTX 4090 and on a 16-chip TPU v5e slice.
  2. end-to-end adaption studies: dense retrieval, CLIP on CoCo and then Flicker30k, and three-recipe multimodal adaption of Jina v5 Omini Nano.
  3. Fixed-work multi-accelerator scaling on 1-8 A100 GPUs.

Overall, the paper is well written and easy to follow.

Critical strengths
  • This paper provides a unified framework for text and multimodal representation learning, across different tasks, such as dense retrieval, similarity ranking, reward modeling, multimodal contrastive learning, masked language modeling, and joint-embedding prediction. Under the framework, it is easy to experimenting different learning methods for different tasks. Reducing the workload required when switching learning objectives.
  • Validation experiments are solid. It compares thirteen paired workload configurations on GPUs and TPUs, end-to-end adaptation studies for dense retrieval and CLIP, and fixed-work strong scaling.
Critical weaknesses
  • While the abstraction layers that Representax presented are well thought, it did not provide analysis or experiments on how different design can affect the choices of components and how specific design choices are made, what are different design choices to make, etc.
  • The scientific contribution of this paper limited for the ICLR main track. It does not show new research enabled by this framework. A more framework/system focused venue (i.e. MLSys) could be a better fit.
Questions for authors
  1. Could you provide an example of changing new objective or model extension, and show specific changes to be made to reuse the overall training, evaluation, and execution lifecycle?
  2. Could you compare the framework to established JAX baseline for dense retrieval, CLIP or other tasks?
1 comment

Thank you for these questions!

  1. Section 3.5 now shows how we added a trainable projection to MPNet. We wrote a model wrapper and a loader, then selected the embedding size and training objective in the config. Across all 18 runs, we reused the same training, evaluation, and checkpoint code. We also checked that training resumed from a checkpoint and that each exported model reproduced its outputs on the test inputs. Appendix I.3 includes the full code we added.

  2. We added a comparison against Hugging Face’s Flax CLIP in Appendix I.1 using the same batches, objective, and optimizer. Flax was faster in all five FP32 runs. BF16 was closer: Flax had higher median throughput (812 vs. 783 examples/s), while Representax was faster in two of the five runs. These measurements cover the training step and exclude preprocessing, evaluation, and checkpointing.

Reviewer_HwmdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Representax, an open-source JAX framework for representation-learning research across modalities. The framework is organized around models/processors, learning tasks, data distributions, and evaluators, connected through a shared training, evaluation, checkpointing, and export lifecycle. It also separates scientific parameters from execution parameters such as sharding, microbatching, precision, and prefetching. The authors implement a set of methods covering retrieval, multimodal contrastive learning, reward modeling, and joint-embedding predictive learning. They compare training performance with established frameworks on GPU and TPU, train and evaluate retrieval and multimodal models, and measure multi-GPU scaling. The paper reports throughput comparisons, held-out retrieval results, and scaling measurements.

Critical strengths
  • The framework is coherent and relatively complete for JAX-based representation-learning research. It covers multiple representation-learning paradigms and algorithms, so it should be useful to researchers already working in the JAX ecosystem.

  • The empirical evaluation is solid. Paired comparisons with established implementations across GPU and TPU settings provide useful evidence about training behavior and execution efficiency, and I find the framework’s usability and reliability convincing.

Critical weaknesses
  • The framework-design novelty is limited. The paper emphasizes separating scientific parameters from execution parameters, organizing experiments around reusable abstractions, and providing a shared lifecycle. These are reasonable choices, but each is already common in modern ML frameworks. I do not see a clear new framework-design principle, or a previously difficult problem that Representax fundamentally addresses. At present, the contribution reads more as a careful integration of established practices than as a new, broadly generalizable insight.

  • Given the limited novelty, acceptance would need to rest on clear practical value to the research community, which is also not sufficiently established. I cannot clearly separate the efficiency gains from the framework design itself from those coming from JAX/XLA. In the TorchInductor control the paper reports, compiled PyTorch is not far from Representax, so the comparison does not show that JAX is a better foundation than PyTorch for future representation-learning frameworks. Considering that PyTorch remains the mainstream framework in representation learning and that most new algorithms will continue to be built on it, without evidence of JAX’s advantage over PyTorch, the cost of switching frameworks would confine the value of Representax largely to the JAX developer community, limiting its contribution.

Questions for authors
  • The paper presents Configuration, Core Abstractions, and Shared Lifecycle as central design elements, but I find these close to established framework practice. Given the ecosystem cost of adopting JAX, could the authors clarify which framework-design or practical deployment insights in Representax specifically depend on JAX, or would be costly to implement under PyTorch? This affects my assessment of the work’s practical contribution.
1 comment

Thank you for the question! The JAX-specific part is how we use the same model parameters for differentiation and sharding. The practical benefit for someone already working in JAX is that they can add a model and reuse the training, evaluation, and checkpoint code. In Section 3.5, we show exactly what we added to MPNet to run it with two existing objectives.

The shared training and evaluation interfaces could also be built in PyTorch. We haven’t measured what that would cost or shown a benefit to switching an existing PyTorch project over. We also added a comparison against Flax CLIP in Appendix I.1. With the same inputs, objective, and optimizer, Flax was faster in both precisions. Our case for Representax is the infrastructure it provides to researchers working in JAX.

Reviewer_RYMBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Representax, an open-source JAX framework for representation-learning research across modalities. It organizes experiments around models and processors, learning tasks, data distributions, and evaluators, and explicitly separates scientific parameters from execution parameters such as sharding and microbatching. It supports retrieval, reward modeling, multimodal contrastive learning, masked language modeling, and joint-embedding prediction. Validation has three parts. First, paired throughput comparisons over 13 workloads against established implementations show higher median throughput in 11/13 configurations on a single RTX 4090 and 13/13 on a 16-chip TPU v5e, though the method reaches only 0.964x against a compiled TorchInductor baseline. Second, end-to-end adaptation experiments show held-out retrieval gains for dense text retrieval and CLIP, and a multimodal adaptation study exposes a trade-off between media gains and text retention. Third, single-node strong scaling of a 505M ModernBERT reaches 85.9% parallel efficiency on 8 A100 GPUs.

Critical strengths
  1. The experimental reporting is unusually rigorous and transparent for a framework paper. Throughput comparisons are paired with matched model, data, objective, batch, and precision settings across five seeds. Configuration bugs found in earlier reference runs are documented and corrected rather than hidden. Negative results (late-interaction regression, BERT slowdowns on GPU, 0.964x against TorchInductor) and an explicit evidence-scope table are reported. This matters because the value of an infrastructure paper depends on whether its performance and correctness claims can be trusted, and here the claims are carefully bounded and largely verifiable.

  2. The framework offers practical breadth with a coherent design. It unifies retrieval, reward modeling, multimodal contrastive learning, MLM, and JEPA-style training in JAX across both GPU and TPU. Its separation of scientific and execution parameters is demonstrated concretely by the fixed-work scaling study, where only execution settings change across device counts. This matters because it could lower the engineering barrier for multimodal embedding research in the JAX ecosystem, which is the paper’s main potential contribution to the community.

Critical weaknesses
  1. The paper is heavily engineering-oriented, and its research novelty and contribution are limited; it reads more like the release documentation of an open-source project than a research paper. The core design ideas (modular abstractions, typed configuration, separating scientific from execution parameters, GradCache, DDP/FSDP sharding) are sound but are mostly well-established engineering practices rather than new abstractions or techniques. The end-to-end experiments only show that standard recipes train successfully, and they yield no new scientific insight. The central claim, that new ideas can be implemented with minimal additional code, is never empirically demonstrated through a new method or a usability comparison. Framework papers previously accepted at ICLR typically offer either system contributions at substantial scale [1] (e.g., TorchTitan), a new abstraction filling a clear ecosystem gap [2] (e.g., TorchRL's TensorDict), or new findings enabled by the tool [3] (e.g., CAX). This paper does not clearly meet any of these, which is the main reason for my recommendation. The authors should demonstrate a new method or finding enabled by the framework, as in [1–3]

  2. The paper’s ambitious framing is not matched by the evidence it provides. The introduction and related work position Representax around natively multimodal learning, the Platonic Representation Hypothesis, and world models. However, the experiments are mostly short throughput runs and modest fine-tuning of existing pretrained models, dominated by text retrieval, and the multimodal and predictive components receive comparatively limited validation. In addition, the throughput gains are measured mainly against eager PyTorch and PyTorch/XLA reference integrations, so they speak more to the efficiency of those particular references than to a clear advantage of the framework itself. This gap between motivation and evidence makes it hard to judge how much the framework actually advances the research directions it claims to enable. The authors should align the framing with the actual evidence.

References:

[1] TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining. ICLR 2025.

[2] TorchRL: A data-driven decision-making library for PyTorch. ICLR 2024.

[3] CAX: Cellular automata accelerated in JAX. ICLR 2025.

Questions for authors

Please see weaknesses.

1 comment

Thank you for your feedback!

  1. We agree that the paper’s main contribution is the software. We haven’t shown a new method or scientific finding enabled by Representax. The example we added in Section 3.5 addresses a narrower question you raised about how much code an extension takes. Appendix I.3 shows the full change we made to MPNet and which parts of training and evaluation we could reuse. We’ve narrowed the paper’s claims accordingly.

  2. Our vision for Representax extends beyond this paper. We revised the introduction and conclusion to make clearer which parts of that vision motivate the work and which workflows we actually evaluated. We also added compiled comparisons in Appendix I.1. Representax was faster on four of six additional TorchInductor workloads, slower on both BERT workloads, and slower than Flax CLIP in both precisions.

Reviewer_hzP4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Representax is a JAX framework that unifies models, learning tasks, data pipelines, and evaluation for representation learning. It supports retrieval, multimodal contrastive learning, reward modeling, and predictive objectives through shared training infrastructure. The evaluation includes thirteen paired GPU and TPU workloads, end-to-end adaptation studies, and fixed-work scaling to eight GPUs. Results show competitive throughput and useful retrieval improvements, while the paper documents configuration differences and limitations.

Critical strengths

1.The software contribution has clear practical value. Separating scientific choices from execution settings, together with shared data and checkpoint interfaces, provides a coherent foundation for experimenting across objectives and modalities. 2.The evaluation is unusually transparent for a framework paper. It includes repeated runs, held-out retrieval quality, a compiled baseline, and an unsuccessful adaptation result. The fixed-work scaling study reaches 6.87 times speedup on eight GPUs, supporting the usefulness of the shared execution infrastructure within the tested setting.

Critical weaknesses

1.Most GPU comparisons use eager PyTorch. In the compiled dense-retrieval control, Representax reaches 0.964 times the reference throughput, so the larger eager-relative gains should not be interpreted as a general advantage over optimized alternatives. The paper acknowledges this, but evidence against compiled references remains narrow. 2.Appendix A reports systematic differences in some TPU loss trajectories, especially cross-encoder and image-text training, without a controlled diagnosis. This leaves uncertainty about whether the throughput comparisons correspond to sufficiently equivalent optimization behavior. The native adaptation studies show useful learning, but do not resolve cross-framework equivalence.

Questions for authors

1.Beyond the dense-retrieval control and Appendix F diagnostics, are any matched compiled-reference measurements available for the other workloads? 2.What checks establish objective and optimizer-update equivalence for the TPU workloads with differing loss trajectories? Are the remaining differences understood?

1 comment

Thank you for these questions!

  1. For additional compiled-reference measurements, we added these to Appendix I.1. We added six workloads with five seeds per arm: cross-encoding, CLIP, MPNet similarity/classification, and BERT similarity/classification. Native/TorchInductor throughput ratios are respectively 1.247, 2.415, 1.274, 1.296, 0.603 and 0.658. Representax is faster on four of the six additional TorchInductor workloads and slower on both BERT workloads. It also trails the original compiled dense-retrieval baseline and the Flax CLIP baseline in both precisions. These results show that our earlier gains over eager PyTorch do not consistently carry over to compiled references.

  2. We have not resolved why the original TPU runs produced different loss curves. We checked both implementations locally using the same model weights and inputs, and separately checked the optimizer and learning-rate schedule. The FP32 results closely agree, but differences remain in BF16. We now state this limitation in Section 4 and report the checks in Appendix I.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.