- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DanLing NestedTensor, a PyTorch tensor abstraction that carries multiple ragged axes (including non-leading axes and N×N or N×M pair states) as part of the tensor itself. Values are stored in a packed buffer together with size, offset, dimension-order, and row-split metadata. Operator handlers, covering 404 operators through __torch_function__ and __torch_dispatch__, create ragged axes through broadcasting, preserve them through feature transformations, and consume them through reductions, so model code does not manage offsets. The representation also supports autograd and torch.compile, with the structural schema fixed at compile time and sizes passed as runtime inputs. Experiments on a single A100 with randomly initialized models measure step time and peak memory. Against padding, the paper reports geometric-mean speedups of 2.74× (eager) and 3.39× (compiled) on BERT and 1.97× on FCN, and 2.40–4.32× on a four-block Pairformer-style workload, which compiles only under DanLing. Additional controls compare against attention-only packing and explicit packing, and separate the interface overhead (≤1.6%) from the gain of batch-level segmented execution.
Critical strengths
1. A clean, composable abstraction with substantial engineering. Ragged axes are created by broadcasting, preserved by feature transformations, and consumed by reductions, and the representation also works through autograd and torch.compile. As a result, multi-ragged model code (e.g., the pair block in Fig. 3) needs no offset bookkeeping. This is the paper's main contribution and what a user would actually gain from it.
2. Honest and well-controlled performance analysis. The matched-granularity controls separate the interface overhead (≤1.6%) from the execution-strategy gain, and the bucketing experiment shows how much of the padding cost can be recovered without the method. Failed cells are reported rather than hidden. This makes it clear where the gains come from, even though it also shows that they come mostly from existing kernels rather than from the representation.
Critical weaknesses
1. The contribution is incremental: the representation does not solve the hard problem. Efficient ragged computation ultimately depends on kernels that natively handle ragged inputs, and the paper does not address this. The paper's own data shows that the gains come from existing kernels rather than from the representation. The interface itself adds ≤1.6% overhead and no speedup, while the 1.5–3.9× gain on the Pairformer workload comes from batch-level segmented calls into an external kernel library (Table 7). Without such kernels, execution falls back to per-sample calls: the non-leading and rectangular regimes gain only 1.19–1.32× over padding and match explicit packing. Representing multi-ragged tensors and mapping a subset of ops to dense kernels is therefore a limited step beyond existing ragged representations.
2. The evaluation does not target workloads that actually need the method, and accuracy is never validated. BERT, GPT-2, the WMT Transformer, and ViT need only one ragged dimension, which existing tools already support. For models that genuinely need two ragged dimensions, mainstream practice is already efficient. Segmentation training typically crops to fixed sizes, which the paper does not compare against, and on DETR the method is about 24% slower than conventional padding (Table 5). The method also changes the computation relative to the standard pipelines of these vision models, yet all models are randomly initialized and no accuracy is reported. A focused study on models that current tools cannot handle well (e.g., real pair-representation models), with both accuracy and performance evaluation, would be more convincing than a general claim of ragged-tensor support.
Questions for authors
See my concerns in the Weakness section.
We have revised the paper to include the complete pretrained AlphaFold2 model and PointNeXt-XL, and to explain how their variable-size representations interact with shared tensor operations and existing kernels. On one A100 with BF16, AlphaFold2 inference reaches 1.81× padded throughput with 65.1% lower peak allocated memory; PointNeXt-XL reaches 1.32× padded throughput. The revised method section also illustrates the MSA, pair, and point-cloud structures used by these models.