- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Null-A, which propagates affine solution sets through a program's Jacobian or its transpose, including when stepwise Jacobians aren't invertible. It reports speedups over CG on small transformer models.
Critical strengths
The method handles rectangular and singular stepwise Jacobians, so inverse AD no longer requires the number of live variable to stay constant. Affine space slices with deferred updates give a concrete implementation of this. Returning the full affine solution set lets users choose a solution by another criterion, e.g. sparsity, which point solvers like CG cannot.
Critical weaknesses
- The only GPU baseline is CG on . The authors attribute the largest gains to ill-conditioning, but no condition numbers are reported. Table 1-3 report no residuals for the Null-A's origin or basis while Table 1 reports CG residuals. CG's tolerance and iteration cap are also not specified, so it's unclear whether the methods produce equally accurate results. Comparisons with LSQR/LSMR and a direct QR/SVD method that also returns the affine solution set would help.
- The paper doesn't show when the full preimage is worth the cost. The GPU experiments form dense block Jacobians and use small models. In the GPT-2 width sweep, Null-A's advantage shrinks (42x faster at width 8, 2x faster at width 32, but actually 2.25x slower than CG at width 40). No memory usage or timing breakdown (e.g. Jacobian vs. preimage computation) is reported. The prompt-editing example also doesn't compare against solving the sparse problem directly without forming the preimage.
Questions for authors
- For each setting in Table 1-3, what are the relative residuals of Null-A's returned origins and basis? What stopping tolerance and iteration cap did CG use?
- For the largest GPT-2 case (width 40), what's the peak memory usage? How is the runtime split between constructing the block Jacobians and computing the preimage?
Final Thoughts
We thank the reviewers and AC for engaging and their insightful comments. The focus on benchmark comparisons with existing methods, however, has allowed the conversation to drift far away from the truly novel contribtion of our submission, stated clearly in the title and abstract. The Null-A method computes the full preimage, and does so in a compositional fashion that operates by a single overloaded execution of the program.
CG, LSQR, and LSMR are point solvers and do not compute the full preimage. They are incomparable, as they do not yield the same result. QR and SVD are not compositional, and not matrix-free. They require explicit computation and representation of the full Jacobian for the entire program, which is infeasible for large programs because it grows as O(mn) space for programs ℜn→ℜm. Null-A is a novel method, we hope even the first instance of a new class of affine-space-based methods, and should be evaluated as such rather than in a benchmark comparison framework.