acceptodds
Under review as a conference paper at ICLR 2027

Statically Typed Tensor Programming with Semantic Dimension Names

Abstract

In mainstream programming languages, types can state what values mean and constrain the operations permitted on them, making code easier to comprehend and enabling automated verification of logical consistency. However, common numerical and machine learning libraries forgo these benefits by representing all multidimensional data with generic tensor types. We present DimWit, a numerical and machine learning library in Scala 3 that encodes tensor dimension names as programmer-defined types, such as Batch, allowing tensor types like Tensor2[Batch, Feature, Float32] for a batch of feature vectors, and extending the benefits of static typing to expressing, tracking, and verifying dimension names and data types in tensors and across their operations. DimWit leverages JAX as its computational engine, inheriting hardware acceleration, just-in-time compilation, and automatic differentiation. To explore DimWit's semantic tensor types in practice, we implement diverse applications, ranging from linear regression to variational autoencoders, as well as clustering algorithms and simulations. We highlight excerpts of these implementations as examples, showing how semantic tensor types give component interfaces precise meaning and can make programs more comprehensible. To demonstrate the type-safety benefits, we reconstruct twelve selected axis- and shape-related defects from widely used open-source projects, each of which silently produced incorrect results, and show that DimWit rejects them at compile time. To confirm that the approach adds negligible runtime cost at scale, we train a 124M-parameter GPT-2 from scratch with model and training code written in Scala 3 and execution delegated to JAX, reaching 97% of the throughput of an equivalent JAX implementation in Python. DimWit illustrates how semantic tensor types can make tensor programs more comprehensible and reject axis- and shape-related defects at compile time, without sacrificing runtime performance. Code available at https://anonymous.4open.science/r/dimwit-9234 and https://anonymous.4open.science/r/deepwit-DC06/.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
named tensor dimensions, static typing, tensor programming, type systems, tensor shape errors, JAX

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kwThOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors provide a library DimWit (along with DeepWit) written in Scala3 to explicitly name the axes of tensor objects for both compile-time and run-time type checking, and to aid in code clarity for programmers. Data in tensor form is the bread and butter of machine learning algorithms, and many errors are introduced by acting over the wrong axes of tensors. These errors are often silent due to similarly size arrays or broadcasting operations, but this library makes them explicit to prevent these errors. Currently these axes are noted by code comments, but this is laborious, unenforced, and frequently out-of-date.

Critical strengths

S1: The authors target a real problem, and their solution for both compile-time and run-time (to my understanding) is a welcome improvement.

S2: The authors present examples of bugs in various libraries where static type checking of tensor axes would have prevented the error.

S3: The authors have many examples of DimWit, including a nice n-body example where they have to relabel the axes, and an LLM which achieves comparable speed to an equivalent JAX implementation.

Critical weaknesses

W1: The DimWit library requires explicitly-sized and labeled tensors, but there are legitimate use cases where it would be useful to have flexibility. Here are two such cases:

W1.1: The authors give an example in line 294 of an "overly general implementation" of a linear layer that operates on the last layer of an abitrary tensor. This feels like a straw-man argument because there is a middle ground between all layers explicitly labeled and no layers labeled. In jaxtyping this might be implemented in the following way,

affine(x: Float[Array, "... in"], A: Float[Array, "out in"], b: Float[Array, "out"]) -> Float[Array, "... out"]:

This allows for arbitrary leading batch axes, but the relevant axes that are being operated on are made clear.

W1.2: Sometimes the number of axes depends on the dimension of the data, but we would like the dimension to also be an argument. For example, a general convolution over D dimensions can be written that might take 2D images with height and width, or 3D images with height, width, and depth. In jaxtyping I often write Float[Array, "Batch Channels Spatial"] to indicate that all the spatial axes come next, although this is not a great solution either.

W2: The library is written for Scala3, which is far less used for machine learning than Python. The authors fully acknowledge this limitation, and valuable research is not limited to the current most popular tools or methods. The authors do not need to address this weakness in their rebuttal, I am just noting it here.

W3 (Suggestion): You discuss the GPT-2 DimWit implementation in the abstract and the introduction, but the results are only in the appendix and not in the main paper. If you give something top billing (i.e. mention it in the abstract) then it should be in the main paper in my opinion. Of course the details can be in the appendix, but the results (Table B.1) and a small description/discussion should be in the main paper. I also think its a nice result.

Questions for authors

Q1: Pertaining to W1.1, is it possible to leave axes unspecified when they are irrelevant to the code functioning?

Q2: Pertaining to W1.2, is it possible to have dynamic numbers of axes?

Q3: What is meant by "theory-aligned design" and "theory-aligned cost function"? Writing down a model f as a function f(x;\theta), a loss function as L(\theta; X,y) is nothing special and makes me think I am missing something with these buzzwordy "theory-aligned" descriptions.

2 comments

W1.1 / Q1 (leaving irrelevant axes unspecified). This is not a straw-man argument. The difference is subtle, yet critical. An affine transformation in mathematics maps a vector to a vector, and the implementation should reflect this. Extra dimensions are baggage that obscures this, and a signature that includes them cannot rule out their use. An illustrative example from practice: deep learning models are often sample-independent, yet batch dimensions are carried throughout the architecture. We implemented GPT-2 and all its modules without a batch dimension in DeepWit, which guarantees that no module can depend on it. The batch dimension appears only in the training loop, i.e., X.vmap(Axis[Batch])(model). This separates concerns: batching, a training concern, does not leak into the architecture. To answer Q1 directly: yes, DimWit can also express a signature with unspecified leading axes using generic types. We deliberately chose not to use it. When a function should apply to more general tensors, we prefer an explicit lifting function as its own concept. For example (data types omitted for brevity):

// Lifts a vector-to-vector function to a matrix-to-matrix function (over the given axis).
def liftOver[B](axis: Axis[B])(f: Tensor1[In] => Tensor1[Out]): Tensor2[B, In] => Tensor2[B, Out] =
  t => t.vmap(axis)(f)

// Composing liftOver with affine gives the batched version; the two concepts stay separate.
def affineBatched(t: Tensor2[Batch, In]): Tensor2[Batch, Out] =
  liftOver(Axis[Batch])(affine(_, A, b))(t)

W1.2 / Q2 (dynamic number of axes). Yes, this is possible. Tensor shapes in DimWit are Scala 3 tuple types, so a type parameter can stand for any number of axes, and Scala provides type-level operations to concatenate such tuple types. DimWit uses this throughout, e.g., its distributions accept tensors of any shape.

W3 (GPT-2 results). We agree. We moved the results (now Table 2, formerly Table B.1) and a short discussion into the main text as the new Section 6, "Run-time performance at scale".

Q3 ("theory-aligned"). We mean that the code has the same structure as the mathematical formulation: the cost is an explicit function costFn: Params => Tensor0, and its gradient is obtained by transforming that function into dCost: Params => Params. This contrasts with, e.g., the widely used PyTorch, where one computes a loss value and calls loss.backward() to calculate the gradients. Neither the cost function nor its derivative ever exists as a function in such code. The functional, theory-aligned style itself comes from JAX and libraries built on it, such as Flax. Semantic types make such code easier to follow because they help keep concepts apart and verify data flow (Section 4.2).

We hope that these clarifications have resolved the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating their score.

Reviewer_kwThOpenReview ↗

Response to the Rebuttal

Thank you for addressing my comments and questions. I find your proposed solution for W1.1 reasonable, if cumbersome. However, that is often the tradeoff with using more strongly-typed languages, as you are advocating for here. I am glad that you are able to define dynamic numbers of axes for W1.2 as that is a valuable use case. I think this work still has some limitations, but on the whole I think deserves acceptance so I am raising my score from a 3 to a 4.

Reviewer_njmMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces DimWit, a Scala 3 tensor library that represents semantic dimension names as types, allowing declarations such as Tensor2[Batch, Feature, Float32]. The compiler tracks these labels through tensor operations and function calls, helping detect mistakes such as swapped dimensions before execution. Its contribution is the integration of this approach with JAX, including automatic differentiation and vectorization, supported by application examples, defect reconstructions, and a GPT-2 training benchmark showing throughput close to plain JAX.

Critical strengths
  • The paper addresses a practical source of errors in ML programming: confusing tensor dimensions with different semantic roles.
  • Static dimension types make function interfaces more informative and allow the compiler to detect axis-related mistakes before execution.
  • The JAX integration keeps runtime overhead small in the tested compiled setting, achieving approximately 97% of plain JAX throughput in the GPT-2 benchmark.
Critical weaknesses
  • Limited novelty.
    Static typing of semantic tensor dimensions already exists in earlier Scala work, including Chen (2017), and in Python’s TensorAnnotations. Jaxtyping also provides dimension-annotated interfaces and runtime shape and dtype checking, although it does not provide DimWit’s static semantic-axis guarantees. The contribution therefore lies mainly in the Scala 3/JAX integration, broader propagation of dimension types, and practical demonstrations. The conceptual advance over existing approaches appears limited.

  • Unclear additional checking coverage over TensorAnnotations.
    TensorAnnotations already supports static semantic-dimension checking for selected JAX operations. The paper argues that DimWit provides broader coverage, including preserving types through grad and vmap, but does not systematically quantify this advantage over TensorAnnotations. The selected defect comparison with jaxtyping does not establish the improvement over an existing static approach. Consequently, it remains unclear how much additional checking capability justifies adopting Scala and its toolchain within predominantly Python-based ML workflows. (The paper acknowledges that Python and not Scala is the predominantly ML / deep learning language.)

  • Limited research advance for ICLR.
    The paper primarily contributes programming abstractions, type safety, and library integration. Its examples and benchmarks demonstrate practical feasibility, while the additional research insight beyond existing dimension-typing approaches remains limited. This emphasis appears more naturally aligned with an ML systems or programming-languages venue, like MLSys.

  • Limited technical detail on the typing mechanism.
    The paper outlines the architecture and illustrates DimWit’s benefits, but delegates the specific type-level logic largely to the source code. In particular, the generic typing rules underlying transformations such as grad and vmap are not developed in detail. This makes it difficult to assess how the stated guarantees are established and which capabilities depend on Scala’s additional type-system expressiveness. For a contribution centered on library design and implementation, these details are needed to evaluate the technical advance. This is especially important, in the light of the comparison to TensorAnnotations and the issues here with grad and vmap.

Very Minor Presentation Issue:
I would suggest to capitalize the first letter whenever you write Section 2, Figure 4...

Questions for authors
  1. Comparison with TensorAnnotations:
    Which of the 12 defect reconstructions rejected by DimWit would TensorAnnotations fail to reject statically under comparable annotations and function interfaces? Please distinguish missing implementation support from limitations of Python’s standard type system, particularly for grad and vmap, and illustrate any claimed expressiveness limitations with concrete examples.

  2. Technical novelty:
    Beyond the Scala 3/JAX integration, which technical capabilities or static guarantees extend those provided by Chen (2017) and TensorAnnotations? For grad and vmap, please distinguish missing implementation support from limitations of Python’s standard type system.

2 comments

The reviewer mentions limited novelty as a main weakness. We disagree. DimWit refines, extends, and combines many ideas shown to be relevant in related work into one full-fledged, easy-to-use library. ICLR directly invites software contributions, and we think a ready-to-use, statically typed, named tensor library with competitive speed is both novel and a direct contribution to ICLR and the broader ML community.

W2 / Q1 (checking coverage over jaxtyping and TensorAnnotations). We added both to Table A.1: each catches 5 of the 13 shape-related defects, compared with 12 for DimWit. TensorAnnotations misses 6 of the 7 due to more than just missing implementations; for example, an operation like reshape can’t be made type-safe without breaking JAX’s API. In DimWit, we replace reshape with more restrictive options like flattening two axes, where the resulting name can be automatically inferred. We acknowledge the cost of adopting Scala and its toolchain in Limitations (now Section 7.2). Python's advantage lies in its ecosystem and the momentum of its adoption, not in the expressiveness of its type system, which is what the checking above depends on.

Q1 / Q2 (grad and vmap: missing implementation vs. type system limitation). Simple cases are a matter of implementation: the type-level logic of jit and grad is expressible in today's Python type system. Python's limitation arises when a type must be computed from other types, since it has no type-level functions. Even sum over one axis has this problem: its return type is the input type without that axis. vmap adds to this, as the function it takes must accept the tensor's type without the mapped axis. TensorAnnotations works around this with one overload per argument combination. Its author lists the resulting combinatorial explosion as one reason for discontinuing the library (Rahtz, 2023). The number of overloads grows combinatorially with tensor rank and axis position, so the missing implementations (i.e., combinations) reflect this underlying type system limitation. In Scala, a single type-level function lets the compiler compute the result type for each concrete case.

Minor: we now capitalize Section, Figure, etc. throughout.

We hope that these clarifications have resolved the reviewer's concerns.

Thank you for the detailed response and the revised manuscript. The new comparison with TensorAnnotations and jaxtyping addresses my main question about checking coverage. Appendix C.6 also makes the technical contribution clearer by showing how DimWit preserves named tensor structure through repeated differentiation.

The core idea of static semantic axis typing is established, and the selected examples do not isolate the effect of axis types from that of stricter APIs. Still, the revision provides enough evidence of the library’s practical value for me to raise my rating from weak rejection to weak acceptance.

Reviewer_xqR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DimWit, a Scala 3 library that makes tensor dimension names part of the static type system. Instead of treating tensors as generic arrays, dimensions such as Batch, Feature, or Context become programmer-defined types that are propagated through tensor operations. This lets the compiler catch certain dimension mismatches before execution, while also making function signatures more descriptive.

The implementation uses JAX as its computational backend through ScalaPy, so the authors retain JIT compilation, automatic differentiation, and hardware acceleration. They demonstrate the approach across several applications, including linear regression, clustering, simulations, VAEs, and GPT-2. The evaluation reconstructs 13 real-world tensor bugs, of which 12 are rejected at compile time. They also train a 124M-parameter GPT-2 model and achieve approximately 97% of the throughput of a comparable JAX implementation.

The broader idea is that tensor programs should expose more of their intended meaning through types, rather than leaving everything implicit in axis positions, comments, and documentation. I think this is an interesting direction, particularly as ML codebases become increasingly complicated and harder to reason about :)

Critical strengths

I really appreciate that the authors built a complete system rather than stopping at a type-system proposal :) Supporting automatic differentiation, vmap, and JIT compilation while achieving around 97% of JAX throughput on GPT-2 is a meaningful engineering result.

I also like how the dimension types make implicit assumptions explicit. The attention and broadcasting examples show why distinguishing dimensions such as Batch and Feature can catch bugs that ordinary tensor shapes might miss. This seems particularly useful as ML systems become larger and increasingly generated by coding agents.

Critical weaknesses

My main concern is what exactly is being verified here :) DimWit checks consistency between programmer-defined dimension labels, but it does not necessarily verify the mathematical meaning of the computation. For example, the n-body simulation explicitly relabels Velocity to Position, but this does not establish that the correct integration was performed. This distinction is particularly relevant when considering stronger verification frameworks, such as Lean-based verified ML systems including TorchLean. I think the paper could be clearer about the boundaries of its guarantees.

Second, I am not convinced that all 12 detected defects specifically demonstrate the benefit of semantic dimension names. Some appear preventable through ordinary static rank typing or more restrictive APIs, such as cases 03, 08, 09, and 10. A simple rank-typed baseline would help isolate the actual contribution. The comprehension claims also remain largely illustrative, and the novelty relative to Chen (2017) could be clearer.

Questions for authors
  1. Type safety vs. semantic correctness: What happens when a tensor is incorrectly labeled or explicitly relabeled? Could DimWit's annotations eventually connect to stronger verification frameworks, such as Lean/TorchLean, where mathematical properties of tensor computations can also be checked?

  2. Defect evaluation: How many of the 12 detected bugs genuinely require semantic dimension names rather than static rank checking or more restrictive APIs? I think a small ablation here would be particularly informative :)

  3. Program comprehension: Is there any quantitative evidence that DimWit makes programs easier to understand or debug? The examples are intuitive, but I wonder how this scales to larger codebases with more complicated dimension vocabularies.

  4. Novelty: What is the main technical advance over Chen (2017), which already explored encoding tensor dimension names as Scala types? Is it primarily the JAX integration, higher-order transformations, or something new in the type-level design?

  5. Static guarantees across JAX boundaries: How are dimension types preserved or validated when moving between Scala and JAX, particularly through ScalaPy, dynamic shapes, or externally supplied tensors? Could an incorrectly typed boundary silently undermine guarantees elsewhere in the program?

3 comments

Thank you for the thoughtful review and for recognizing what we care about most: that DimWit is a complete system, not just a type-system proposal. We revised the paper to address your two main concerns: clarifying what DimWit guarantees and comparing these guarantees to the related work as a baseline.

W1 / Q1 (what is verified). The compiler verifies only whether the program is consistent in its dimension names and data types. DimWit does not aspire to be a system for formal verification, such as TorchLean. Instead, it provides similar advantages over JAX or PyTorch as statically typed languages provide over dynamically typed ones: namely, it helps reduce the number of bugs and, more importantly, aids our understanding by letting us express concepts represented by the tensors as types without guaranteeing correctness. We have now made this point more explicit in the related work.

W2 / Q2 (defect evaluation). We implemented all shape-defect cases in JAX with jaxtyping and tensor_annotations and added the results to Table A.1. jaxtyping captures 5 at runtime, tensor_annotations 5 at compile time, compared with 12 in DimWit. Five cases (01, 05, 07, 12, 13) are caught only by DimWit. We'd expect a rank-only checker to perform similarly to these two baselines overall.

Q3 (program comprehension). We don't have quantitative evidence, as noted in Limitations (new Section 7.2). However, in other domains, statically typed languages are widely seen to lead to better code abstractions and overall higher code quality. We'd expect the same to be true of DimWit. Furthermore, we'd expect the vocabulary of dimension types to help teams and communities find a shared language of concepts.

W2 / Q4 (novelty over Chen 2017). While the core ideas behind the named tensor dimensions are already there in Chen 2017, DimWit makes program comprehension the primary goal, extends type safety to transformations like vmap, higher-order differentiation, named tensor shapes, and tensor trees, and develops the idea into a full-fledged library, with JAX as a high-performance computational engine. We clarified this in the related work section.

Q5 (JAX boundary). We wrap the JAX methods in type-level logic in DimWit to preserve names correctly. User-defined functions use these base DimWit functions (like they would use JAX), so all types are preserved (i.e., tracking composes). If the user requires manual wrapping (e.g., a custom Python function written in JAX), the type-level tracking must be implemented in that wrapper, which the compiler then trusts.

We hope that these clarifications have resolved the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating their score.

Reviewer_xqR1OpenReview ↗

Response to the Authors

Thanks for the detailed response and for taking the time to run the additional comparisons! :) I appreciate the new Table A.1, especially the five defects caught only by DimWit, and the clarification of the relationship to Chen (2017).

However, I still have some reservations about the central contribution. While the new baselines show that DimWit catches more defects, they don't quite isolate how much of this comes specifically from semantic dimension names rather than the broader type system and API restrictions. I also appreciate the authors being upfront about the lack of quantitative evidence for improved program comprehension, but since this is presented as a primary motivation, I think that evidence matters. The JAX boundary is another limitation worth being clear about. As the authors explain, custom wrappers must manually implement the dimension tracking, and the compiler trusts those annotations. This is a reasonable design choice, but it does limit the guarantees in practice.

Overall, I think DimWit is a useful engineering contribution, and I appreciate the authors' response. My concerns are less about whether the system works and more about whether the evaluation sufficiently establishes its distinctive benefits over existing approaches. For these reasons, I'm maintaining my original score! I hope the authors find these comments helpful! :)

Thank you for following up and for the very interesting discussion points.

The question of whether the semantics of the names matter, or only the tensor type improves correctness, is, in our opinion, the crucial link between better understandability of the programs and improved correctness. Going back to our quantitative evaluation, we could catch exactly the same number of bugs by choosing dimension names with no semantic meaning, like "A0" and "A1" instead of "Frame" and "Channel". The type system would guarantee that we use them consistently. However, only once we choose names that make semantic sense (i.e., correspond to concepts in the domain we care about) will they provide any value to us in program comprehension.

The same is not true when we have no names, but only the rank encoded in the type. To isolate the guarantees from dimension names over static rank checking under the same API, we ran one more ablation. We simulate a rank-only baseline for all 12 cases DimWit rejects by using DimWit and manually casting tensors after each operation to a single per-rank type (e.g., Rank1). Here is a minimal code example:

type Rank1 = Tensor1[A0, Float32] // One type for all Tensor1
type Rank2 = Tensor2[A0, A1, Float32] // One type for all Tensor2
val t: Rank2 = …
// The resulting Tensor1[A1] is explicitly cast to the monolithic Rank1 type
val t2: Rank1 = t.sum(Axis[A0]).asInstanceOf[Rank1]

In this rank-only DimWit simulation, 4 of them are still rejected at compile time (02, 03, 09, 10), and the other 8 are missed. For example, case 08 mentioned in the original review is missed in rank-only DimWit because the output tensor type does not track the fused names that DimWit tracks through the flatten operation (see added Appendix C.9 for details of this operation).

Regarding the JAX boundary: The intended way to use DimWit is by using the typed operations that DimWit provides. However, in practice, it may happen that a special functionality a user needs is not available, or that for some other reason they want to bypass the type checking. Being able to access the underlying JAX-Tensors serves as an escape hatch in such cases, which, as you point out, limits the guarantees that we can give.

Comments on the paper

1 comment

We thank the reviewers for their careful reading of our paper and the insightful comments. We have uploaded a revised version of the manuscript, which addresses the mentioned concerns, adds a new baseline that illustrates the advantage over the existing python libraries in finding bugs and better clarified the novelty over the closest related work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.