- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to organize LLM outputs for Theoretical research in computer science or mathematics. The authors note that current AI-generated results in this area are usually shared in the form of a paper that contains only the ideas that worked, without necessarily sharing the ideas that did not work, the partial results towards a better or more general proof etc. This paper proposes to explicitly constrain the model to organize his research in the form of an AND/OR directed acyclic graph, which encodes the claims/lemmas/theorems used and the dependencies among those results. This additional structure can then be shared to help other people or AI agents to build on previous attempts without trying the same things over and over. The authors apply this idea to 2 separate problems in theoretical computer science. On these problems, they show that their approach induces only a small token overhead, while allowing for the AI models to find mistakes more reliably compared to just giving the flawed proof.
Critical strengths
-This paper studies a timely subject, given the recent results of AI generated proofs.
-The problems studied are interesting
Critical weaknesses
-The experiments are quite narrow on 2 specific problems. Results would be more compelling with more repetitions of this strategy on other problems.
-It is unclear what is precisely "paper + chat + evidence" compared to "paper + DAG + evidence" proposed by the authors. How is "paper + chat + evidence" constructed exactly ?
-Minor weakness: some sentence are really unclear, for instance in abstract "Existing projects release supporting artifacts organized around research runs rather than individual claims," was very unclear to me before reading the rest of abstract.
Questions for authors
- Have you tried your framework on other problems ? Another thing that could be great is if your framework could be used to explain some of the recent math results by OpenAI more clearly. Have you tried your framework to explain some results?
Dear Reviewers,
The authors have responded to your comments. Could you please review their responses and update your reviews?