acceptodds
Under review as a conference paper at ICLR 2027

hyperpath: Deep Learning Programs as Directed Hypergraphs

Abstract

Research code in machine learning is written twice. The first version is the idea, usually a diagram of values and the relationships between them. The second decides the loops over time, what to keep and for how long, how to batch, and where to compile. That second version takes the effort and carries the bugs, and when a language model writes it, someone still has to read it and check that it is right. We present hyperpath, a Python library in which a program is a directed (possibly cyclic) hypergraph. Each edge relates a set of values through an ordinary Python function and says nothing about when it runs. The user states which values they hold and which they want. The library compiles the one (acyclic) derivation the graph admits, naming what it could not reach where there is none and declining to choose where there are two. Users write time as a local relationship, x[t] and x[t+1], and the library derives each call's loop over time, how long it keeps each value and how much of the sequence runs at once, leaving the user to declare a loop's direction and the order of an update. We evaluate by reproduction, writing GPT-2 124M against llm.c's PyTorch trainer, ResNet-50 against torchvision's ImageNet recipe, DDPM against its CIFAR-10 reference and PPO and DQN against CleanRL. We re-run every reference on the same hardware, and each graph reproduces its reference's result at comparable cost. We then write nineteen questions and variants of three of these models and of a VAE, a GAN and nanoGPT. The diffusion model's reference writes its loop over the timesteps by hand four times, for sampling, for DDIM, for the images predicted along the chain and for the likelihood, where its graph writes no loop and derives all four.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
machine learning systems, declarative programming, programming models, vectorization, reproducibility, PyTorch

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5sFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces hyperpath, a Python library that represents deep learning programs as directed hypergraphs. The core motivation is to eliminate the "written twice" problem in machine learning research code: researchers first conceptualize the mathematical relationships and then must manually implement the engineering details. In hyperpath, users only need to declare local relationships between values through ordinary Python functions without specifying execution timing. Given the user-specified known inputs and target outputs, the system automatically derives an acyclic computation path via backward search and infers the global scheduling strategy. The authors evaluate the system by reproducing five canonical models and comparing them against highly optimized reference implementations, demonstrating consistency in fidelity and comparable computational cost. Furthermore, through evaluations across 19 variants and interpretability queries on three models, a VAE, a GAN, and nanoGPT, the authors demonstrate that this approach significantly reduces code modification size (diff size) while entirely eliminating the need for hand-written temporal loops or forward hook injections.

Critical strengths
  1. The system design is rigorous and the motivation is well-founded. By formulating time as a local relationship within a directed hypergraph, the system can automatically derive complex scheduling mechanisms without requiring users to explicitly write control flow.
  2. The evaluation is highly rigorous. By re-running the exact, vendored reference implementations on the same hardware with identical random seeds, and demonstrating bit-for-bit consistency on complex queries alongside statistically indistinguishable final metrics, the authors convincingly prove that the hypergraph derivation introduces no hidden semantic errors or numerical drift.
Critical weaknesses
  1. Lack of Baseline Comparisons with Existing Systems. Although the authors compared the hyperpath implementation with native PyTorch implementations, the paper positions itself as a software system. Therefore, to prove the necessity and effectiveness of hyperpath, the focus of the comparison should be on existing software systems, such as JAX. Since these existing systems already possess relevant functional optimizations, what are the specific advantages of hyperpath compared to them?
  2. Concerns Regarding Usability Evaluation. The usability evaluation in this paper seems limited to reducing code volume to make it easier for LLMs to generate and humans to review. In reality, usability encompasses other critical aspects, such as whether the interface is similar or consistent with mainstream frameworks, and whether existing code can be easily migrated to hyperpath. The current framework defines many specific keywords, and users must pay close attention to numerous implementation details; otherwise, silent computational errors will occur. This significantly diminishes its overall usability.
  3. Limited Generalization. While not a fatal flaw, the current hyperpath design appears heavily overfitted to dense tensor operations common in deep learning. It seems to lack support for other data structures, such as sparse graphs, and currently does not support distributed systems. However, this could be addressed as a future vision.
  4. Lack of a System Architecture Diagram. While this is a systems contribution rather than a purely theoretical one, it would be much more appropriate to provide a system architecture diagram rather than exclusively pasting code. A system architecture diagram allows users to intuitively understand the system's goals and design, whereas pasting code extensively makes the paper read more like a development manual than a research paper introducing a novel system.
Questions for authors
  1. Have the authors considered comparing their approach with current code-generation agents? As code-generation agents become increasingly powerful, can providing appropriate prompts also achieve program optimization and handle temporal loops? This raises a critical question: does hyperpath merely introduce an additional set of DSL interfaces/abstractions without fundamentally outperforming agent-assisted PyTorch coding?
  2. Could the authors provide experimental results comparing hyperpath with existing software systems to further substantiate the systemic advantages of hyperpath?
3 comments

Thank you for the review and the concrete suggestions. Point by point:

"the focus of the comparison should be on existing software systems, such as JAX ... what are the specific advantages"

§6 now has a comparison table (Table 2, p. 10) covering PyTorch, TensorFlow 1, Theano, JAX and Tempo, and we checked the marks, most of them by running each system. For JAX in short: you write every loop yourself. lax.scan takes the step function, the direction and what to carry, and lax.while_loop (the counterpart of our until=) can't be differentiated in reverse mode. JAX never works out which part needs a loop or what to keep in memory. The DDPM code loops over the denoising steps four times by hand (sampling, DDIM, a strip of predicted images, the likelihood), and in JAX it would too. In hyperpath all four come from one graph with no loop written. For the strip, the graph needs no new function, just the call (72 tokens against the reference's 248).

"Could the authors provide experimental results comparing hyperpath with existing software systems"

We didn't add a JAX port, and we'd rather say why than skip the question. A JAX version would mostly measure XLA against torch.compile, and our port against our own reference, not the idea. Our cost claim is narrower: does deriving the program cost anything against the reference as its authors ship it? On the same hardware and seeds, the graphs take 0.67 to 1.00 of each reference's time (Table 1). PPO's 0.67 comes from how CleanRL's file is written. Against the same arithmetic written by hand, the graph takes 1.03 to 1.04. The differences in Table 2 are about what each system can do, and a timing wouldn't show them.

"whether the interface is similar or consistent with mainstream frameworks, and whether existing code can be easily migrated"

The code inside each function stays ordinary PyTorch, and existing models are used unchanged. The DDPM graph calls the reference's UNet as it is, ResNet-50 is torchvision's model, and GPT-2's functions call llm.c's modules. On short runs each graph ends with exactly the same weights as its reference (for PPO and DQN, a copy of its loop), bit for bit. What's new is how the pieces are connected. We agree that's a new set of words to learn. Appendix A lists all of it and which experiments use what.

"users must pay close attention to numerous implementation details; otherwise, silent computational errors will occur"

Some declarations can change a result without an error, yes, but "numerous" overstates it. There are four, and §7 lists all of them, for example an after= ordering an update that no value depends on. §7 also gives a check: build the same program again with everything run one step at a time (lifting="never") and compare. On our graphs the two agree to within 1e-5 of each output's largest value, or bit for bit. The comparison caught both errors we planted on purpose. If you had other cases in mind, we'd like to know which.

"overfitted to dense tensor operations ... sparse graphs, and currently does not support distributed systems"

We agree the evaluation covers dense tensors on one process, but not that the design is tied to them. Which functions run, and in which loops, depends only on which values depend on which, not on what the values are. A step's value can be anything the functions accept: a micro-batch from a data loader, an environment's state, a tuple. Storage is a plug-in per backend, so sparse storage would be a new backend. As §7 says, we plan for one process and leave multi-device work to the backend.

"it would be much more appropriate to provide a system architecture diagram rather than exclusively pasting code"

We added one: Figure 2, which opens Appendix A (p. 17). It shows what the user writes, what hyperpath derives for each question, and where it hands work to existing code such as torch.compile. The ten pages have no room left, so it sits at the head of the appendix, and Figure 1's caption points to it.

"does hyperpath merely introduce an additional set of DSL interfaces/abstractions without fundamentally outperforming agent-assisted PyTorch coding?"

The evaluation already contains this comparison, though we should have said so more clearly. A coding agent wrote every answer that §5.3 counts, on both sides, except the three DDPM methods the reference already had. So §5.3 compares agent-written PyTorch with an agent-written graph, under the same rules. The graph is shorter on 14 of 19 questions and PyTorch on five, which we report. None of the graph's answers is a loop over time or a hook into a model. An agent can write the loops, sure. But someone still has to check them. In PyTorch each new question is another program with its own loop that has to agree with the others. In the graph it's a few lines and a call against the same model, and the planner refuses a question it can't answer in exactly one way.

While some of my initial concerns have been addressed, critical reservations remain regarding usability and system-level evaluation. First, as a systems contribution, the framework should ideally employ static analysis or IR-level runtime assertions to prevent silent mathematical errors. Shifting the burden of semantic verification onto the users is an unacceptable design trade-off for a declarative DSL. Second, my concern regarding baseline comparisons extends beyond merely measuring token counts. A rigorous systems evaluation must also quantify system efficiency and validate the correctness of the derived implementations by demonstrating competitive downstream task performance against existing declarative frameworks, such as JAX.

Thank you for the follow-up. We think the paper already answers part of both points, so we'd like to show where, and say plainly where we see it differently.

"the framework should ideally employ static analysis or IR-level runtime assertions to prevent silent mathematical errors"

It does both. Before anything runs, the planner refuses a question it can't answer (naming the value it couldn't reach), a question with two possible programs (printing both), an ordering cycle, and two side-effecting functions that nothing orders (§2.2, §4.1, §7). While it runs, every function lifted to a whole sequence checks its shapes on every call and raises instead of broadcasting into a wrong result (§3), at 1 to 6% of eager wall-clock on the CPU (§7).

What these checks can't catch are the four declarations §7 lists. Each one is about what a user's own Python function does that no value records: whether it works elementwise, whether it has side effects, or whether it changes state in place. Whether an arbitrary Python function has such a property can't be decided by static analysis in general. JAX is in the same position: jit and vmap assume the function is pure and leave that to the user, and a custom_vjp is trusted unless the user tests it. So we don't think this trade-off is specific to hyperpath. What hyperpath adds is a second program for the same question: derived with lifting="never", it runs every function one step at a time, and comparing the two caught both errors we planted (§7). If you have an assertion in mind that would catch one of the four without the user's help, we'd like to hear it.

"quantify system efficiency and validate the correctness of the derived implementations by demonstrating competitive downstream task performance"

That is what Table 1 (p. 8) reports; the token counts are only the third of our three claims. Downstream results against each model's reference implementation, re-run on the same hardware:

  • GPT-2 124M: validation loss 3.2773 against 3.2772 after 10B tokens, HellaSwag 0.301 against 0.302.
  • ResNet-50: top-1 75.49 against 75.61 after 90 epochs on ImageNet.
  • DDPM: FID 3.236 against 3.225 on CIFAR-10.
  • PPO and DQN: returns over ten seeds per environment (thirty on HalfCheetah), where every difference's interval contains zero.

For efficiency, the graphs take 0.67 to 1.00 of each reference's wall-clock, and §5.2 reports throughput and peak memory beside it.

"against existing declarative frameworks, such as JAX"

Our baselines are widely used reference implementations: llm.c, torchvision's recipe, CleanRL and the DDPM reference. A JAX version of the same model would mostly compare JAX's compiler, XLA, with PyTorch, one backend with another. hyperpath isn't a compiler: it builds each question's program, which runs on PyTorch and goes to torch.compile where asked. And in JAX the user writes each loop (lax.scan, lax.while_loop) and says what it carries. hyperpath derives the loops and what each keeps from the steps each function reads, and the user declares only which way a loop runs. Table 2 compares the two on what each system does for the user.

Still, we ran one such comparison where a JAX version already exists and isn't ours: CleanRL's own dqn_jax.py beside its PyTorch dqn.py, on our three DQN environments, ten seeds each, on the same machine. Returns matched (IQM 498.3 against 497.9 on CartPole, −89.5 against −89.0 on Acrobot, −190.9 against −197.7 on MountainCar, all inside each other's 95% intervals), and the JAX file took 0.99 to 1.01 of the PyTorch file's time. Our graph takes 0.98 to 0.99 of that same PyTorch file's time (Table 1), so here it matches the JAX version too. These are small tasks, so this says nothing about large models, where XLA's compiler may matter more.

Reviewer_SVYPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a code analysis framework called hyperpath. It presents an example and uses it throughout the paper to describe the steps conducted in the framework. Code in this graph can be “executed,” and this is compared to running the original code in terms of runtime.

Critical strengths
  • Generally a reasonable paper structure
Critical weaknesses
  • Extremely difficult to understand the content of the paper due to the English language
  • Paper is described along an example, including the methodology sections that should contain some algorithm
Questions for authors

Introduction: It does not explain or motivate the problem well. Generally, one may not start with a diagram --- at least not in a literal sense. Perhaps a white-board drawing, a sketch, or plainly an idea in mind. But one may not call this writing research code, i.e., programming. It is ideation and conceptualization.

One understands that the proposed hyperpath framework begins when code exists, in Python, operating on the control flow (shown in Figure 1). Data flow and a combination of both would be the other views.

One does not understand why the example is a hypergraph? For example, there is a connection between "project" and "recur" in Figure 1 except that it correlates to have x as input. So, in the graph, there is basically an edge from x --> project and x --> recur. As one would expect in control flow.

Almost all parts of the paper are not semantically understandable, e.g., paragraphs 3 and 4 of the introduction.

Section 2 describes an example rather than the algorithm conducting the backward search on the graph to pick a route and a plan. A similar statement applies to Section 3. Later claims on complexity (see Limitations) cannot be verified due to the absence of methodological details.

The evaluation (Section 5) is done, as the authors state, "by reproduction". A graph-based representation is "executed" versus running the original code. What it basically does is conduct some form of unit test: whether the code is properly reflected in the graph. So it is more a form of testing the framework rather than a scientific evaluation.

Limitations: It is more an extension of experimentation with new results, e.g., on planning and ambiguity.

The authors are honest to state that their framework is in its infancy, with no external users or documented uses. This is not an argument against the paper, which itself can have a good research contribution.

My recommendation is to clean up the paper in terms of language to make clear what the hyperpath is actually about. Add methodological/algorithmic details and choose tools/algorithms to compare with (some are mentioned in the paper). Then take a workshop at a graph conference and submit as early work or work in progress.

4 comments

Thank you for the review. We think several of the points below come from reading hyperpath as a tool that analyses existing code, which it isn't. We answer each point and ask a few questions back where we couldn't tell what to change.

"Extremely difficult to understand the content of the paper due to the English language" "Almost all parts of the paper are not semantically understandable, e.g., paragraphs 3 and 4 of the introduction."

We want to fix this, but these two sentences don't tell us what to fix. Could you point us to a few sentences that were hard to follow, and say whether the problem is the grammar or the terms? The other two reviews summarize the paper accurately, so we suspect the terms more than the English. We rewrote the abstract and the introduction in plainer words, paragraphs 3 and 4 included. If parts of the new version are still unclear, please tell us which.

"One understands that the proposed hyperpath framework begins when code exists, in Python, operating on the control flow (shown in Figure 1)."

That's not what hyperpath does, and the summary ("a code analysis framework") rests on it. Nothing is analysed. The user writes the graph, and the code in Figure 1 is the whole model. The graph is data flow, not control flow. Each vertex is a value (a tensor, or a tensor at every time step), and each square box is an ordinary Python function that computes some values from others. Figure 1's caption now says this.

"One does not understand why the example is a hypergraph? ... there is basically an edge from x --> project and x --> recur"

project and recur aren't values, they're functions (the edges). x is a value, and both functions read it. It's a hypergraph for two reasons. One function can compute several values at once: project computes q, k and v together, the way one linear layer does, which an ordinary edge can't express. And a value can have more than one function computing it: h comes from attend or from recur. A question (which values you have, which you want) picks one route through these choices.

"one may not start with a diagram ... It is ideation and conceptualization."

We agree a whiteboard sketch isn't code, and the revision no longer says it is. But that's our point: the values and how they depend on each other are what a researcher already has in mind, and with hyperpath that part is the program.

"Section 2 describes an example rather than the algorithm ... Later claims on complexity (see Limitations) cannot be verified"

Sections 2 and 3 show what the user writes, and an example is the clearest way to show that. The algorithm belongs in §4, and it's there now: definitions and Algorithm 1 in §4.1 (pp. 6-7), the full search as Algorithm 2 in Appendix B (pp. 26-31), and proofs of how hard the problem is. In general, finding a plan is NP-hard, and checking that there's only one is coNP-hard. In a simpler case both are fast.

"What it basically does is conduct some form of unit test ... more a form of testing the framework rather than a scientific evaluation."

We disagree. A unit test checks a small piece against a known answer. We trained full models next to their published code: GPT-2 on 10 billion tokens, three times beside llm.c, and ResNet-50 for 90 epochs on ImageNet. Each reaches its reference's result at about the same speed (Table 1). §5.3 then measures what nineteen new questions cost to write. If hyperpath is a new way to write models, the questions are whether it gives the same models, at what cost, and with how much less code. Which evaluation would you have expected instead?

"Limitations: It is more an extension of experimentation with new results, e.g., on planning and ambiguity."

We're not sure what you mean here. The section lists what doesn't work yet or costs time, with numbers so the reader can judge. For example, the DQN graph plans in 12 ms and would take 8.4 s without the shortcuts in §4.1. Should the numbers move to §5, or is the criticism something else?

"take a workshop at a graph conference and submit as early work"

hyperpath isn't a contribution to graph algorithms. The graph is how a user writes a machine learning program, and the evaluation trains ML models against their reference code. That's why we submitted to ICLR's area for infrastructure and software libraries. On your other suggestions, we did all three: plainer language, Algorithm 1 and Appendix B, and Table 2 in §6, which compares hyperpath with PyTorch, TensorFlow 1, Theano, JAX and Tempo.

Thank you again for your review. Since the discussion closes soon, here is where the revised PDF follows your three recommendations:

  • plainer language: rewritten abstract and introduction
  • methodological details: definitions and Algorithm 1 in §4.1, the full search in Appendix B
  • comparison with other tools: Table 2 in §6 (PyTorch, TensorFlow 1, Theano, JAX, Tempo)

If parts are still hard to follow, pointing us to a sentence or two would help a lot.

Reviewer_SVYPOpenReview ↗

Clarity of the paper

I see Algorithm 1, Table 2, etc. While the table shows the functional differences, it does not make the overall benefits of the work clearer. I follow the other reviewers that a thorough comparison of the proposed tool with the existing approaches needs to be done. This may also include a user study?!

As you asked about what text could be clearer, see, e.g., 5.1 CLAIMS AND PROTOCOL. I tried myself to improve the language by asking a language model to rephrase the first paragraph. It helped a bit, but overall it did not allow to understand really what you intend to say. A similar statement can be made about basically all other paragaphs, take another example: "We run each reference in the mode and precision its authors ship, and the graph in the same, compiled only where the reference compiles itself. " -- What does mean? Especially the itself-part as in "compiled only where the reference compiles itself"?

Since this paper is an infrastructure or systems paper, a demo would also be appreciated. That would help. A video is also often used to demonstrate system.

Given the current status, I maintain my score.

Thank you for reading the revised paper and for the concrete examples. They help us.

"compiled only where the reference compiles itself"

"The reference" is the original code that we compare with. Some original code uses the PyTorch compiler (torch.compile), and some does not. If the original code uses the compiler, we also use it for hyperpath. If the original code does not use it, we also do not use it for hyperpath. In this way, both sides run under the same conditions, and the speed comparison is fair.

We agree that Section 5.1 is too dense. Here is a simpler version of its first paragraph:

We test three claims.

  • C1, fidelity: A model written as a hyperpath graph reaches the same result as the original implementation, re-run on our hardware.
  • C2, cost: The graph runs about as fast as the original implementation and, where we record it, needs about the same memory, on the same hardware and with the same numerical precision. We compile the graph with torch.compile only if the original implementation is compiled. Where we additionally compile a graph whose original implementation is not compiled, we compare that run only with the uncompiled graph, so the compiler's speed-up is never credited to hyperpath.
  • C3, size of a change: A new variant of a model, or a new question about it, requires only new edges in the graph, or none, rather than a separate program. We measure this as the size of the code change against the original implementation.

We will rewrite the other paragraphs of the paper in the same simple way.

"it does not make the overall benefits of the work clearer"

Researchers often write a separate program for each task with the same model: to train it, to generate samples, to generate samples in a second way, to compute its likelihood, or to look inside it. For example, the original DDPM code writes its loop over the denoising steps four times by hand: for sampling, for DDIM sampling, for the images it predicts along the way, and for the likelihood. With hyperpath, the researcher writes the model only once. For each task (the paper calls it a "question"), hyperpath builds the program automatically.

Section 5 shows two results:

  • There is almost no extra cost. The results match the original code, and hyperpath takes between 67% and 100% of the time of the original code (Table 1).
  • New tasks need less code. We tested 19 tasks and variants. In total, hyperpath needs 2,754 tokens of code and PyTorch 3,632, and hyperpath is shorter for 14 of the 19. None of the hyperpath answers is a loop written by hand (Section 5.3).

"a thorough comparison of the proposed tool with the existing approaches needs to be done. This may also include a user study"

Table 2 compares the features of the systems. Table 1 compares with the original code that researchers use today. In our answer to Reviewer 5sFh, we also compared with CleanRL's JAX version of DQN. We did not do a user study. Section 5.3 measures the size of each change, but not how long a person needs to write it. We cannot add a user study in this revision.

"a demo would also be appreciated"

The supplementary material contains the library and examples that you can run. A small demo is examples/basics/derive.py. It builds one graph, asks three questions, and prints the program that hyperpath builds for each question. examples/README.md lists all examples. We cannot make a video before the discussion ends, but we agree that a video would help.

Reviewer_47XkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue that (particularly in machine learning code) the same content is "written" repeatedly, first in abstract form and then in concrete form. It studies a tool that handles this problem in a unified manner: the user defines a computation graph that may contain ambiguity, and the tool derives concrete computation procedures by inference over that graph. Different derivation for the same variable and time-series changes are incorporated as elements of the graph, which is then expanded into a strict computation graph based on additional conditions given by the actual problem setting. The paper shows that the generated programs are equivalent to the reference implementations up to numerical error, and that modifying a program to answer additional requests requires only a small amount of change.

Critical strengths
  • The idea of working backward from the desired values to identify a route through an ambiguous graph, or to raise an error otherwise, is interesting and I personally like it. Merging variables whose identical meaning is guaranteed (by the researcher) into one vertex of the graph avoids redundant descriptions, and since the experiments confirm this numerically (as the number of changed tokens), it can be regarded as a concrete advantage.
  • The experimental results are reported in a straightforward manner. The paper mentions potentially arbitrary experimental decisions and what appear to be negative results in a way that allows readers to check them, which I appreciate in an experimental study.
Critical weaknesses
  • I understood the proposed method to be essentially a variant of a lazily evaluated computation graph, differing in that it allows multiple derivations for the same variable. Apart from the part concerning multiple routes, much of what is listed as the contributions is already addressed by computation graph libraries (in particular, libraries such as Theano and TensorFlow v1 whose default is to build a static graph). Loops along the time dimension are simply treated on a computation graph as an unrolled DAG with initial values, so it is hard to claim anything beyond a difference in representation. For variable-length loops, though there are already specialized implementations such as Tempo [1], but the essential problem in this case is that the loop's continuation condition is determined by the values of the variables themselves. This is my subjective view as an engineer, but handling this directly with a dynamic computation graph or an ordinary program (with appropriate parts of static graphs) is more intuitive for most people and allows ad-hoc intervention inside the loop, so I question the benefit of deliberately introducing the proposed method as an intermediate representation (the until condition mentioned in the Questions imposes a considerably higher cognitive cost than a simple Python loop, and unfortunately does not seem well suited to writing complex processing). Allowing multiple derivations for a variable helps avoid problems such as the combinatorial explosion that arises when every variant is defined separately. However, it brings out another drawback: it becomes difficult to tell whether variables that share a name but have different derivation histories are also semantically identical.
  • The work is interesting as a way of implementing computation graphs, but in practical use it is unclear whether it resolves the situation that "research code in machine learning is written twice" as in the Introduction. It is quite rare to define every variable in a computation graph by oneself; even researchers usually rely on other people's implementations for efficiency and numerical accuracy, often without being able to choose how those implementations are written. As a result, values and computations that don't exist in the graph one currently has in-mind get processed, and conversely, variables that do exist in the graph get skipped. From the viewpoint of computational efficiency as well, it would be hard to claim practicality unless the method is maintained so that it can always compile into efficient programs. That would require continuous updates of the method's own backend, which seems quite challenging given that modern deep learning libraries have become large-scale development projects.

[1] Pedro F. Silvestre and Peter Pietzuch. "Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs." In Proc. SOSP, 2025.

Questions for authors
  • The proposed method has an argument until that handles the stopping condition of variable-length loops. I'd like to know how far it can support complex loops that go beyond a simple condition check. Also, must horizon always be specified together with until? If so, is horizon merely an upper bound that does not impose any additional implementation constraints?
  • Although the paper does not mention it explicitly, the procedure of expanding the proposed graph into a concrete program brings to mind topics such as the determinization of nondeterministic automata and mode analysis in compilers. I think results from the field of computation theory could be applied within the scope of the proposed method as well. For example, existing algorithms could likely be leveraged to efficiently decide the existence and ambiguity of derivations.
  • If there were a mechanism that stored a variable's derivation process as an attribute of the variable itself, so that it could be taken into account when deriving other variables, the issue raised at the end of Weakness 1 might instead become an advantage of an extended version of this method. I don't think such a mechanism exists at present, but personally I find it an interesting orientation.
  • If the method could handle outputs after infinite loop iterations (i.e., solving fixed points of recursive functions), that would be an additional advantage over DAG-based computation graphs. Within the scope of this paper, however, the derivation result seems to be simply unrolled into a DAG. If there is potential for such an extension, please add a discussion.
  • (I have also used AI partially in this review, so I am in no position to be too critical, though) the main text read to me as a list of facts, and I did not sense a coherent story in the paper. I believe that text whose main claim is unclear can be fixed if humans supervise it carefully (or rather, I personally consider it important that this is ensured), so it would be good to put more effort into this aspect.
2 comments

Thank you for the careful review, and for pointing us to Tempo, which we should have cited. We're glad you like the backward search for a route, or an error otherwise.

"much of what is listed as the contributions is already addressed by ... Theano and TensorFlow v1"

Right, TF1's session.run and theano.function already run only what a request needs (§6 now says so). Three things they don't do:

  1. Choose between producers. Each tensor has one producing op, so an alternative becomes a tf.cond or a second graph. In hyperpath h has two, and each call picks one.
  2. Work out the loop. In TF1 you write each loop, and training and generation need different ones. From the same seven functions, our Figure 1 graph (with attention) gives a training program with no loop and a generation program with one.
  3. Reach inside a loop. TF1 can't fetch a value computed inside a while_loop, so DDPM's strip of predicted images needs a second loop (248 tokens in the PyTorch reference). Our graph needs just the call (72).

"Loops along the time dimension are simply treated ... as an unrolled DAG"

They aren't unrolled. A value over time is at most six vertices for the planner. A loop is checked once at any step and once at the first, so one plan serves every length (§4.1).

"such as Tempo [1]"

Tempo is now in §6, Table 2 and Appendix E. It schedules the backward pass, tiles, swaps memory to the host and handles several time dimensions, which we don't. Where we differ:

  • Tempo derives one schedule per program and settles overlapping definitions by the order written; we build a program per question and refuse one with two answers.
  • What a call is given changes our program. For example, given a prompt, our plan processes it in one batch before the token-by-token loop (the "prefill"). Tempo's own Llama decoder runs the prompt through the loop and leaves the prefill as a TODO.
  • Our functions are unchanged third-party code. Tempo uses its own 44 operators and neither fuses nor differentiates other Python.
  • until= stops a loop on a computed value, which Tempo doesn't do yet.

"the until condition ... imposes a considerably higher cognitive cost than a simple Python loop"

For one question a plain loop is simpler, agreed; the graph pays off once one model answers several. A Python loop around calls still works, and g.watch reads values while a loop runs.

"must horizon always be specified together with until?"

No, it's only a cap. Newton's iteration for √2 stops after six iterates with no horizon or horizon=100, and after three with horizon=3 (Figure 6). The condition is any expression over the step's values, one true/false per step, so a batch of sequences stops all together. If you have a loop in mind that this can't express, we'd like to try it.

"variables that share a name but have different derivation histories"

Giving a value two producers says they compute the same thing, and each call uses exactly one. Storing a value's derivation on it isn't in hyperpath yet, but we'd like to try it.

"It is quite rare to define every variable in a computation graph by oneself"

You don't have to. A function can be anyone's code, and what it computes inside stays inside, as with the reference UNet in the DDPM graph. To reach a value inside someone's model, you split it using its own modules: our GPT-2 graph calls llm.c's layers, two functions per block, so §5.3's interpretability questions need no hooks. And hyperpath writes no kernels; it hands plain Python to torch.compile.

"existing algorithms could likely be leveraged to efficiently decide the existence and ambiguity"

Appendix B now states what we can prove.

  • If every function computes one value (plus three more conditions), your intuition holds: a known linear-time search (Gallo et al., 1993) finds a plan, and rerunning it once per function of that plan finds a second. Our planner doesn't use it yet (§4.1 shows what that costs).
  • Once a function can compute several values, like project computing q, k and v, deciding whether a plan exists is NP-complete even without loops over time, and deciding whether there's only one is coNP-complete.
  • Mercury's compiler builds a procedure per mode, i.e. per choice of which arguments are given; a question is that choice for a graph (Appendix E).

"solving fixed points of recursive functions"

hyperpath reaches a fixed point by iterating: x[t+1] << f(x[t]), with until= stopping once x stops changing, as Figure 6 does for Newton's method. It's one loop, not unrolled, and gradients go back through every iteration that ran. Solving for the fixed point directly and differentiating it implicitly, as deep equilibrium models do (Bai et al., 2019), isn't built in; a user's function can do it.

"I did not sense a coherent story in the paper"

Fair, the abstract and introduction now make one argument (see the general response).

Thank you again for your review. Since the discussion closes soon, here is where the revised PDF addresses your points:

  • Tempo: §6, Table 2 and Appendix E
  • the planning algorithm: Algorithm 1 in §4.1
  • deciding whether a plan exists and whether it's unique: Appendix B. It's fast when every function computes one value, and NP-complete / coNP-complete once a function can compute several.
  • horizon= is only an upper limit: Figure 6 caption

If anything is still unclear or unconvincing, we're happy to answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.