acceptodds
Under review as a conference paper at ICLR 2027

CERTCOMPILE: Preserving Evidence under a Delivery Budget

Abstract

Retrieval agents often read more evidence than they can pass to a downstream reader. Selecting records by relevance alone can omit intermediate facts needed to support an answer. We present CERTCOMPILE, which records source spans and execution dependencies, prioritizes supporting records within a delivery budget, and verifies the evidence before returning an answer. Rejected executions fall back to the original agent. On 120 development questions, a 512-token execution budget and a 256-token delivery budget yield 12 valid accepted answers with dependency-first packing, compared with five for rank-order packing and one for forwarding all admitted evidence. Exact-match accuracy including fallback is 39.17%, 38.33%, and 35.83%, respectively. The policies agree when the budgets are equal. On 240 MuSiQue questions with two models, prioritizing model-generated citation dependencies does not improve answer quality over the same producer's ranking. These results support preserving execution dependencies when execution and delivery budgets differ, while showing that better evidence retention need not translate into comparable gains in answer accuracy or lower inference cost.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM agents, retrieval-augmented generation, evidence preservation, program reuse, runtime verification, budget-constrained inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZjjLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose CERTCOMPILE, a retrieval agent that prioritizes packing evidence into a constrained reader budget. They test it on 120 development questions, and on 240 questions from an existing dataset.

Critical strengths
  • The mismatch of reader/execution budget is a real and interesting problem
Critical weaknesses
  • The introduction does not have a single reference to other works. Likewise the remaining sections of the paper have very few references, and as such, this work reads as wholly unconnected to previous works and literature.
  • There is little to no explanation of the data (e.g. these "120 natural-language development questions" and where they come from)
  • The problem setting is not well explained: the task, metrics, data, and baselines are difficult to locate, which in turn makes the results very difficult to interpret.
  • If I understand correctly, the metric seems circular for Table 2. You are essentially measuring whether all the dependencies are present, and the Dependency-first packing method ensures that those exact dependencies are included. The real test is in fact whether including the dependencies leads to better answer generation, which I think is the null result in Table 3.
Questions for authors

Why is it important to us that the dependencies are included in a fixed budget, given that you show that doing so for QA does not lead to improved answers?

Reviewer_Xg3MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that for retrieval agents, selecting records by relevance can omit the intermediate facts to support an answer. To solve this problem, the authors presents CERTCOMPILE, which records the execution dependencies, prioritizes supporting records, and verifies the evidence. The experiments show that preserving execution dependency improve valid accepted answers under specific budgets, but not gain in downstream answer accuracy.

Critical strengths

The paper aims to address a practical problem in retrieval agent, and design a complete pipeline. The analysis makes sense and supporting experiments are provided.

Critical weaknesses

The paper argues that preserving intermediate dependencies is important for producing reliable answers. However, the results do not clearly demonstrate this. E.g., in Table 5, compared with Compose + Rank, Compose + Dependency indeed has more accepted answers (5->12), but only improves EM from 40.00% to 40.83%. Given that the evaluation consists of only 120 questions, this corresponds to just one more correct.

It seems that the author manually design a verifier detecting dependency, and then propose a packing strategy specifically designed to satisfy this requirement. But this change does not translate into a comparable gain in final answer accuracy.

Finally, the presentation and organization could be improved. There are many awkward sentences and confusing words, and here I just randomly list some of them:

  • In Section 3.1, the author mentions "Resolver v7 binds ..." What is that? Why do the authors mention a version abruptly here?
  • Why is the detail that "the policies agree when the budgets are equal" emphasized in the abstract?
  • The last paragraph lists many limitations, such as, "none of the methods achieves a correct answer in the eight-target blind evaluation" and then ends abruptly.
Questions for authors

Please clarify the weaknesses.

Reviewer_YixNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies evidence delivery for retrieval agents when the evidence available during execution exceeds what can be passed to a downstream reader or interface. CERTCOMPILE records typed retrieval executions, including source spans and dependencies between intermediate results, and prioritizes the supporting records of the terminal value under a separate delivery budget. A verifier checks execution and delivery consistency, with rejected executions falling back to the original agent.

Critical strengths

The paper points out that intermediate connection facts in multi-hop problems also need to be preserved, which is practically significant for systems requiring source auditing, transmission of execution results, or those constrained by downstream context.

In the core packing experiment, all strategies share the same execution results, retrieved evidence, and fallback responses, differing only in their packing methods. This allows for a clearer identification of whether "retaining more complete evidence" stems from the packing strategy itself.

Critical weaknesses

Once an execution already provides reliable dependencies, retaining the terminal value’s transitive support is a relatively direct strategy. The empirical question is therefore what the additional program library, composition machinery, and verification infrastructure contribute. On the main natural-language evaluation, Date-only matches Compose’s decisions, while Direct-Typed and the no-library constructor achieve the same number of jointly valid acceptances. These controls are valuable, but their results weaken the evidence that the broader architecture provides a substantial practical advantage over direct typed execution with dependency-aware packing.

Given that the system has already completed an execution path, obtained a candidate answer, and recorded relevant evidence and dependencies, the packer's task becomes relatively straightforward: to retain the necessary evidence within the delivery budget while avoiding truncating existing dependency chains. However, for more open-ended natural language tasks, accurately identifying relationships, discovering intermediate facts, and establishing reliable dependencies often represent the main challenges. The concern here is the difficulty of obtaining reliable execution dependencies, not whether the system reads ground-truth answers at runtime. The current results show that dependency-based evidence retention is useful with the particular executor studied in this paper, but it is still unclear how well this carries over to broader tasks. I would like to see a representative real-world application showing that reliable execution dependencies are readily available and that the delivery budget is indeed an important bottleneck.

Questions for authors

Can you show a realistic application where reliable execution dependencies are already available and the delivery budget is an important bottleneck? The positive results currently come mainly from a restricted two-hop executor.

Date-only and the no-library constructor achieve the same valid coverage as Compose. What does the program library add over these simpler methods, and when would it be needed?

In the typed experiments, the answer is computed before evidence packing. Does dependency-first packing still improve answer quality when an independent reader must answer using only the packed evidence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.