- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this paper, the authors propose CERTCOMPILE, a retrieval agent that prioritizes packing evidence into a constrained reader budget. They test it on 120 development questions, and on 240 questions from an existing dataset.
Critical strengths
- The mismatch of reader/execution budget is a real and interesting problem
Critical weaknesses
- The introduction does not have a single reference to other works. Likewise the remaining sections of the paper have very few references, and as such, this work reads as wholly unconnected to previous works and literature.
- There is little to no explanation of the data (e.g. these "120 natural-language development questions" and where they come from)
- The problem setting is not well explained: the task, metrics, data, and baselines are difficult to locate, which in turn makes the results very difficult to interpret.
- If I understand correctly, the metric seems circular for Table 2. You are essentially measuring whether all the dependencies are present, and the Dependency-first packing method ensures that those exact dependencies are included. The real test is in fact whether including the dependencies leads to better answer generation, which I think is the null result in Table 3.
Questions for authors
Why is it important to us that the dependencies are included in a fixed budget, given that you show that doing so for QA does not lead to improved answers?