- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors provide a library DimWit (along with DeepWit) written in Scala3 to explicitly name the axes of tensor objects for both compile-time and run-time type checking, and to aid in code clarity for programmers. Data in tensor form is the bread and butter of machine learning algorithms, and many errors are introduced by acting over the wrong axes of tensors. These errors are often silent due to similarly size arrays or broadcasting operations, but this library makes them explicit to prevent these errors. Currently these axes are noted by code comments, but this is laborious, unenforced, and frequently out-of-date.
Critical strengths
S1: The authors target a real problem, and their solution for both compile-time and run-time (to my understanding) is a welcome improvement.
S2: The authors present examples of bugs in various libraries where static type checking of tensor axes would have prevented the error.
S3: The authors have many examples of DimWit, including a nice n-body example where they have to relabel the axes, and an LLM which achieves comparable speed to an equivalent JAX implementation.
Critical weaknesses
W1: The DimWit library requires explicitly-sized and labeled tensors, but there are legitimate use cases where it would be useful to have flexibility. Here are two such cases:
W1.1: The authors give an example in line 294 of an "overly general implementation" of a linear layer that operates on the last layer of an abitrary tensor. This feels like a straw-man argument because there is a middle ground between all layers explicitly labeled and no layers labeled. In jaxtyping this might be implemented in the following way,
affine(x: Float[Array, "... in"], A: Float[Array, "out in"], b: Float[Array, "out"]) -> Float[Array, "... out"]:
This allows for arbitrary leading batch axes, but the relevant axes that are being operated on are made clear.
W1.2: Sometimes the number of axes depends on the dimension of the data, but we would like the dimension to also be an argument. For example, a general convolution over D dimensions can be written that might take 2D images with height and width, or 3D images with height, width, and depth. In jaxtyping I often write Float[Array, "Batch Channels Spatial"] to indicate that all the spatial axes come next, although this is not a great solution either.
W2: The library is written for Scala3, which is far less used for machine learning than Python. The authors fully acknowledge this limitation, and valuable research is not limited to the current most popular tools or methods. The authors do not need to address this weakness in their rebuttal, I am just noting it here.
W3 (Suggestion): You discuss the GPT-2 DimWit implementation in the abstract and the introduction, but the results are only in the appendix and not in the main paper. If you give something top billing (i.e. mention it in the abstract) then it should be in the main paper in my opinion. Of course the details can be in the appendix, but the results (Table B.1) and a small description/discussion should be in the main paper. I also think its a nice result.
Questions for authors
Q1: Pertaining to W1.1, is it possible to leave axes unspecified when they are irrelevant to the code functioning?
Q2: Pertaining to W1.2, is it possible to have dynamic numbers of axes?
Q3: What is meant by "theory-aligned design" and "theory-aligned cost function"? Writing down a model f as a function f(x;\theta), a loss function as L(\theta; X,y) is nothing special and makes me think I am missing something with these buzzwordy "theory-aligned" descriptions.
2 comments
Response to the Rebuttal
Thank you for addressing my comments and questions. I find your proposed solution for W1.1 reasonable, if cumbersome. However, that is often the tradeoff with using more strongly-typed languages, as you are advocating for here. I am glad that you are able to define dynamic numbers of axes for W1.2 as that is a valuable use case. I think this work still has some limitations, but on the whole I think deserves acceptance so I am raising my score from a 3 to a 4.
W1.1 / Q1 (leaving irrelevant axes unspecified). This is not a straw-man argument. The difference is subtle, yet critical. An affine transformation in mathematics maps a vector to a vector, and the implementation should reflect this. Extra dimensions are baggage that obscures this, and a signature that includes them cannot rule out their use. An illustrative example from practice: deep learning models are often sample-independent, yet batch dimensions are carried throughout the architecture. We implemented GPT-2 and all its modules without a batch dimension in DeepWit, which guarantees that no module can depend on it. The batch dimension appears only in the training loop, i.e.,
X.vmap(Axis[Batch])(model). This separates concerns: batching, a training concern, does not leak into the architecture. To answer Q1 directly: yes, DimWit can also express a signature with unspecified leading axes using generic types. We deliberately chose not to use it. When a function should apply to more general tensors, we prefer an explicit lifting function as its own concept. For example (data types omitted for brevity):W1.2 / Q2 (dynamic number of axes). Yes, this is possible. Tensor shapes in DimWit are Scala 3 tuple types, so a type parameter can stand for any number of axes, and Scala provides type-level operations to concatenate such tuple types. DimWit uses this throughout, e.g., its distributions accept tensors of any shape.
W3 (GPT-2 results). We agree. We moved the results (now Table 2, formerly Table B.1) and a short discussion into the main text as the new Section 6, "Run-time performance at scale".
Q3 ("theory-aligned"). We mean that the code has the same structure as the mathematical formulation: the cost is an explicit function costFn: Params => Tensor0, and its gradient is obtained by transforming that function into dCost: Params => Params. This contrasts with, e.g., the widely used PyTorch, where one computes a loss value and calls loss.backward() to calculate the gradients. Neither the cost function nor its derivative ever exists as a function in such code. The functional, theory-aligned style itself comes from JAX and libraries built on it, such as Flax. Semantic types make such code easier to follow because they help keep concepts apart and verify data flow (Section 4.2).
We hope that these clarifications have resolved the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating their score.