- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces hyperpath, a Python library that represents deep learning programs as directed hypergraphs. The core motivation is to eliminate the "written twice" problem in machine learning research code: researchers first conceptualize the mathematical relationships and then must manually implement the engineering details. In hyperpath, users only need to declare local relationships between values through ordinary Python functions without specifying execution timing. Given the user-specified known inputs and target outputs, the system automatically derives an acyclic computation path via backward search and infers the global scheduling strategy. The authors evaluate the system by reproducing five canonical models and comparing them against highly optimized reference implementations, demonstrating consistency in fidelity and comparable computational cost. Furthermore, through evaluations across 19 variants and interpretability queries on three models, a VAE, a GAN, and nanoGPT, the authors demonstrate that this approach significantly reduces code modification size (diff size) while entirely eliminating the need for hand-written temporal loops or forward hook injections.
Critical strengths
- The system design is rigorous and the motivation is well-founded. By formulating time as a local relationship within a directed hypergraph, the system can automatically derive complex scheduling mechanisms without requiring users to explicitly write control flow.
- The evaluation is highly rigorous. By re-running the exact, vendored reference implementations on the same hardware with identical random seeds, and demonstrating bit-for-bit consistency on complex queries alongside statistically indistinguishable final metrics, the authors convincingly prove that the hypergraph derivation introduces no hidden semantic errors or numerical drift.
Critical weaknesses
- Lack of Baseline Comparisons with Existing Systems. Although the authors compared the hyperpath implementation with native PyTorch implementations, the paper positions itself as a software system. Therefore, to prove the necessity and effectiveness of hyperpath, the focus of the comparison should be on existing software systems, such as JAX. Since these existing systems already possess relevant functional optimizations, what are the specific advantages of hyperpath compared to them?
- Concerns Regarding Usability Evaluation. The usability evaluation in this paper seems limited to reducing code volume to make it easier for LLMs to generate and humans to review. In reality, usability encompasses other critical aspects, such as whether the interface is similar or consistent with mainstream frameworks, and whether existing code can be easily migrated to hyperpath. The current framework defines many specific keywords, and users must pay close attention to numerous implementation details; otherwise, silent computational errors will occur. This significantly diminishes its overall usability.
- Limited Generalization. While not a fatal flaw, the current hyperpath design appears heavily overfitted to dense tensor operations common in deep learning. It seems to lack support for other data structures, such as sparse graphs, and currently does not support distributed systems. However, this could be addressed as a future vision.
- Lack of a System Architecture Diagram. While this is a systems contribution rather than a purely theoretical one, it would be much more appropriate to provide a system architecture diagram rather than exclusively pasting code. A system architecture diagram allows users to intuitively understand the system's goals and design, whereas pasting code extensively makes the paper read more like a development manual than a research paper introducing a novel system.
Questions for authors
- Have the authors considered comparing their approach with current code-generation agents? As code-generation agents become increasingly powerful, can providing appropriate prompts also achieve program optimization and handle temporal loops? This raises a critical question: does hyperpath merely introduce an additional set of DSL interfaces/abstractions without fundamentally outperforming agent-assisted PyTorch coding?
- Could the authors provide experimental results comparing hyperpath with existing software systems to further substantiate the systemic advantages of hyperpath?
3 comments
While some of my initial concerns have been addressed, critical reservations remain regarding usability and system-level evaluation. First, as a systems contribution, the framework should ideally employ static analysis or IR-level runtime assertions to prevent silent mathematical errors. Shifting the burden of semantic verification onto the users is an unacceptable design trade-off for a declarative DSL. Second, my concern regarding baseline comparisons extends beyond merely measuring token counts. A rigorous systems evaluation must also quantify system efficiency and validate the correctness of the derived implementations by demonstrating competitive downstream task performance against existing declarative frameworks, such as JAX.
Thank you for the follow-up. We think the paper already answers part of both points, so we'd like to show where, and say plainly where we see it differently.
"the framework should ideally employ static analysis or IR-level runtime assertions to prevent silent mathematical errors"
It does both. Before anything runs, the planner refuses a question it can't answer (naming the value it couldn't reach), a question with two possible programs (printing both), an ordering cycle, and two side-effecting functions that nothing orders (§2.2, §4.1, §7). While it runs, every function lifted to a whole sequence checks its shapes on every call and raises instead of broadcasting into a wrong result (§3), at 1 to 6% of eager wall-clock on the CPU (§7).
What these checks can't catch are the four declarations §7 lists. Each one is about what a user's own Python function does that no value records: whether it works elementwise, whether it has side effects, or whether it changes state in place. Whether an arbitrary Python function has such a property can't be decided by static analysis in general. JAX is in the same position: jit and vmap assume the function is pure and leave that to the user, and a custom_vjp is trusted unless the user tests it. So we don't think this trade-off is specific to hyperpath. What hyperpath adds is a second program for the same question: derived with lifting="never", it runs every function one step at a time, and comparing the two caught both errors we planted (§7). If you have an assertion in mind that would catch one of the four without the user's help, we'd like to hear it.
"quantify system efficiency and validate the correctness of the derived implementations by demonstrating competitive downstream task performance"
That is what Table 1 (p. 8) reports; the token counts are only the third of our three claims. Downstream results against each model's reference implementation, re-run on the same hardware:
- GPT-2 124M: validation loss 3.2773 against 3.2772 after 10B tokens, HellaSwag 0.301 against 0.302.
- ResNet-50: top-1 75.49 against 75.61 after 90 epochs on ImageNet.
- DDPM: FID 3.236 against 3.225 on CIFAR-10.
- PPO and DQN: returns over ten seeds per environment (thirty on HalfCheetah), where every difference's interval contains zero.
For efficiency, the graphs take 0.67 to 1.00 of each reference's wall-clock, and §5.2 reports throughput and peak memory beside it.
"against existing declarative frameworks, such as JAX"
Our baselines are widely used reference implementations: llm.c, torchvision's recipe, CleanRL and the DDPM reference. A JAX version of the same model would mostly compare JAX's compiler, XLA, with PyTorch, one backend with another. hyperpath isn't a compiler: it builds each question's program, which runs on PyTorch and goes to torch.compile where asked. And in JAX the user writes each loop (lax.scan, lax.while_loop) and says what it carries. hyperpath derives the loops and what each keeps from the steps each function reads, and the user declares only which way a loop runs. Table 2 compares the two on what each system does for the user.
Still, we ran one such comparison where a JAX version already exists and isn't ours: CleanRL's own dqn_jax.py beside its PyTorch dqn.py, on our three DQN environments, ten seeds each, on the same machine. Returns matched (IQM 498.3 against 497.9 on CartPole, −89.5 against −89.0 on Acrobot, −190.9 against −197.7 on MountainCar, all inside each other's 95% intervals), and the JAX file took 0.99 to 1.01 of the PyTorch file's time. Our graph takes 0.98 to 0.99 of that same PyTorch file's time (Table 1), so here it matches the JAX version too. These are small tasks, so this says nothing about large models, where XLA's compiler may matter more.
Thank you for the review and the concrete suggestions. Point by point:
§6 now has a comparison table (Table 2, p. 10) covering PyTorch, TensorFlow 1, Theano, JAX and Tempo, and we checked the marks, most of them by running each system. For JAX in short: you write every loop yourself.
lax.scantakes the step function, the direction and what to carry, andlax.while_loop(the counterpart of ouruntil=) can't be differentiated in reverse mode. JAX never works out which part needs a loop or what to keep in memory. The DDPM code loops over the denoising steps four times by hand (sampling, DDIM, a strip of predicted images, the likelihood), and in JAX it would too. In hyperpath all four come from one graph with no loop written. For the strip, the graph needs no new function, just the call (72 tokens against the reference's 248).We didn't add a JAX port, and we'd rather say why than skip the question. A JAX version would mostly measure XLA against torch.compile, and our port against our own reference, not the idea. Our cost claim is narrower: does deriving the program cost anything against the reference as its authors ship it? On the same hardware and seeds, the graphs take 0.67 to 1.00 of each reference's time (Table 1). PPO's 0.67 comes from how CleanRL's file is written. Against the same arithmetic written by hand, the graph takes 1.03 to 1.04. The differences in Table 2 are about what each system can do, and a timing wouldn't show them.
The code inside each function stays ordinary PyTorch, and existing models are used unchanged. The DDPM graph calls the reference's UNet as it is, ResNet-50 is torchvision's model, and GPT-2's functions call llm.c's modules. On short runs each graph ends with exactly the same weights as its reference (for PPO and DQN, a copy of its loop), bit for bit. What's new is how the pieces are connected. We agree that's a new set of words to learn. Appendix A lists all of it and which experiments use what.
Some declarations can change a result without an error, yes, but "numerous" overstates it. There are four, and §7 lists all of them, for example an
after=ordering an update that no value depends on. §7 also gives a check: build the same program again with everything run one step at a time (lifting="never") and compare. On our graphs the two agree to within 1e-5 of each output's largest value, or bit for bit. The comparison caught both errors we planted on purpose. If you had other cases in mind, we'd like to know which.We agree the evaluation covers dense tensors on one process, but not that the design is tied to them. Which functions run, and in which loops, depends only on which values depend on which, not on what the values are. A step's value can be anything the functions accept: a micro-batch from a data loader, an environment's state, a tuple. Storage is a plug-in per backend, so sparse storage would be a new backend. As §7 says, we plan for one process and leave multi-device work to the backend.
We added one: Figure 2, which opens Appendix A (p. 17). It shows what the user writes, what hyperpath derives for each question, and where it hands work to existing code such as
torch.compile. The ten pages have no room left, so it sits at the head of the appendix, and Figure 1's caption points to it.The evaluation already contains this comparison, though we should have said so more clearly. A coding agent wrote every answer that §5.3 counts, on both sides, except the three DDPM methods the reference already had. So §5.3 compares agent-written PyTorch with an agent-written graph, under the same rules. The graph is shorter on 14 of 19 questions and PyTorch on five, which we report. None of the graph's answers is a loop over time or a hook into a model. An agent can write the loops, sure. But someone still has to check them. In PyTorch each new question is another program with its own loop that has to agree with the others. In the graph it's a few lines and a call against the same model, and the planner refuses a question it can't answer in exactly one way.