- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses a setting in which AI models search over programs with the goal of finding executable models that explain observed phenomena (e.g., physics simulations). One approach in this setting is a two-level approach: first, search over program structures; then, for a given program structure optimize continuous parameters with gradient descent. However this can be expensive, since a candidate program must be compiled into a reverse-mode AD graph. The paper proposes a differentiable virtual machine which amortizes the cost of differentiating candidate programs, and an approach to batched execution of a given program to further accelerate search. The paper evaluates on a fluid simulation, finding that the proposed approach successfully finds a program and corresponding parameters that match behavior of held-out data.
Critical strengths
- To the best of my knowledge, the proposed approach is technically sound
- The work has an interesting and important core conceit, that of accelerating the speed at which AI agents can iterate in the task of scientific discovery; doing this via accelerating the inner search in SGA is an interesting approach
Critical weaknesses
- To the best of my understanding, the core contribution of the paper is in the speedup of NDVM vs Jax (and not in the general approach of using gradient descent in the inner loop). However, the evaluation does not actually demonstrate that this speedup leads to more accurate discovery. For example, I would have expected to see a plot showing something along the lines of held-out fitness vs wall-clock time showing that NDVM results in faster optimization than a naive approach.
- While there is some discussion of the performance characteristics in Section 3, it’s not clear how the various different cited wall-clock numbers add up to compare against a well-tuned Jax baseline.
- The paper also does not sufficiently describe NDVM: what are the set of programs that one can write in this language? What are the semantics of these programs? How does the batching executor work with substantially divergent program traces (e.g., looping for different numbers of iterations, nested branches, etc)?
Questions for authors
I would be willing to raise my score conditioned on an evaluation demonstrating that NDVM-based optimization is more efficient than baseline approaches, and on a more complete description of the NDVM language.