acceptodds
Under review as a conference paper at ICLR 2027

CBR-VERIFY: BUDGETED EVIDENCE REVISIONFOR REUSABLE SCIENTIFIC AGENT WORKFLOW

Abstract

Scientific agents must revise the evidence supporting their conclusions when mea-surements, assumptions or evaluation protocols change. CBR-VERIFY formulatesthis decision as budgeted claim completion. Executable predicates define decisions,version-bound obligations define required evidence, and closure selection allocatescomputation between archive validation and conditional repair. The completion ob-jective captures complementary prerequisites and shared acquisition costs; boundedseed enumeration ranks ready actions by covering density. Across 12 paired GPTtasks, full CBR improves observed utility over Plan–Verify by 9.52 percentagepoints while using 7.85% fewer reported tokens and 12.13% more scientific credits.A broader frozen-task replication, repeated executions, and an additional modelpreserve the main advantage. The replicated mechanism comparison gains 15.48utility points over immediate-completion greedy while saving 4.35 scientific creditsper task. Structural sweeps identify stable pair-expansion gains under high sharing,no gain under low sharing, and reproduction on tasks excluded from tuning. Thesame controller remains beneficial against caching, dependency-aware, cost-awarecontrols without retuning on new test outcomes. Real adapter executions validateevidence invalidation and revision; the real-execution comparison also saves ap-proximately 17% of simulation-tool execution time. Independent replay verifieskey trajectories and outcomes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
scientific agents, budgeted planning, shared-cost optimization, reusable scientific workflows

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5HvFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CBR-VERIFY, an evidence-revision and action-selection layer for scientific workflows when upstream data, calibration parameters, or evaluation splits change under a limited computational budget. Rather than recomputing an entire pipeline from scratch or risking stale conclusions, CBR-VERIFY organizes workflow state into a three-tier structure: (1) upstream sources and protocol assumptions, (2) intermediate artifacts bound via recursive SHA256 signatures (σv\sigma_v) over values and parent hashes, and (3) weighted claims c=(ϕc,Rc,sc,wc)c = (\phi_c, R_c, s_c, w_c) gated by executable Boolean predicates that output supported, refuted, or insufficient (whenever any required artifact binding is stale or unverified).

On top of this incremental lineage substrate, the authors formulate budgeted claim completion as a Set-Union Knapsack Problem (SUKP) with supermodular rewards (capturing complementary prerequisites that yield zero single-step gain until full closure) and shared acquisition costs across claims. They also introduce a conditional acquisition rule that decides whether to run a cheap validation probe (vav_a) or directly recompute (rar_a) based on a stale prior pap_a, probing iff (1−pa)ra>va(1 - p_a)r_a > v_a. The resulting Closure Advisor enumerates singleton and pair claim seeds (1≤∣K∣≤21 \le |K| \le 2), ranks ready actions by covering density ρt(K)\rho_t(K), and proposes up to six candidate actions to either a deterministic selector or an LLM controller (gpt-5.6-luna).

Across a 12-task paired GPT comparison, full CBR improves evidence utility over a Plan–Verify baseline by 9.52 percentage points while using 7.85% fewer reported tokens and achieving zero false admissions (0/125). In a 36-task deterministic selector comparison, full CBR outperforms immediate-completion greedy by 15.48 utility points and singleton selection by 5.12 points. On a real-data archive of 61 multichannel thermal runs across five revision sequences, full CBR gains 5.71 utility points over a dependency-aware scheduler at an intermediate budget (11 credits) and saves 18.82% of scientific credits at equal utility (94.29%) under a wider budget (15 credits).

Critical strengths
  1. Principled Budgeted Scheduling over Incremental Lineage (Complementarity, Shared Costs, and Conditional Probing): While DAG provenance and content-addressable caching already exist in build systems (e.g., Make/Bazel, Adapton) and data pipelines (DVC), they typically recompute all invalidated targets or require manual target selection without budget awareness. CBR-VERIFY makes a genuine algorithmic contribution by adding a budget-constrained economic planner on top of this substrate. By formulating evidence revision as Set-Union Knapsack and pricing three coupled mechanisms—(i) whole-closure completion to avoid myopic greedy starvation on multi-step prerequisites, (ii) shared prerequisite cost amortization across paired claims, and (iii) expected-value comparison between cheap invariance probing (e.g., adjacent sensor temperature differences remaining invariant under a uniform calibration offset δ\delta) and expensive recomputation—the method provides a clean, well-motivated design principle for cost-effective verification.

  2. Strict Internal Evidence Solidity and Elimination of Unverified LLM Hallucinations (Zero False Admissions): Scientific agents frequently over-claim experimental success when upstream conditions shift or budgets run out. By enforcing recursive SHA256 parent bindings (At(c)=1A_t(c) = 1) and an executable checker gate that forces any claim with incomplete or stale prerequisites into the insufficient state, CBR-VERIFY guarantees strong internal solidity within the declared contract. Empirically, this eliminates false admissions in the GPT controller experiments (0/125 for Full CBR vs. 5/117 for No-advisor) and shows in Section 5.3 that explicit executable checker feedback—unlike generic verbal reflection—successfully converts unsupported guesses into justified abstentions or corrected labels.

Critical weaknesses
  1. Restricted to Pre-Declared Static Workflows and Fixed Claim Scopes (No Open-Ended Experimental Planning or Cross-Project Reuse): Despite being framed around "Scientific Agent Workflows," CBR-VERIFY operates strictly as a maintenance and scheduling auditor after a human engineer has manually authored a domain adapter specifying the complete dependency DAG G=(V,E)G=(V,E), tool prices, stale priors pap_a, and fixed executable predicates ϕc\phi_c with pre-declared scope scs_c (e.g., checking whether a wall-temperature difference at 30 s is ≤18∘C\le 18^\circ\text{C}). First, this means the system only guarantees local/internal solidity (that the ledger matches the current parameters), not external scientific validity—it cannot evaluate whether a hypothesis generalizes to broader regimes or proactively propose new experiments to expand a claim's scope. Second, when starting a genuinely new research project, CBR-VERIFY cannot help an agent design experiments from scratch, nor can its SHA256 artifact ledger transfer empirical insights across projects unless they share bit-identical source contracts and tool digests.

  2. Reproducibility Gaps (Unrecoverable Raw Execution Records) and Underpowered LLM Statistical Contrasts: Ironically for a paper centered on verifiable ledgers and provenance certificates, the manuscript repeatedly acknowledges that the original task-level execution records for Table 2 (the 36-task deterministic selector comparison) are unavailable for independent reaggregation, relying instead on post-hoc aggregate consistency checks (Appendix E.1), and that access to the 59,475 action-level records for the thermal cohort is currently unverified. Furthermore, in the 12-task LLM controller experiment (Table 1), all 12 prespecified primary comparisons yield Holm-adjusted p≥0.0938p \ge 0.0938, meaning the observed gains over Plan–Verify and No-sharing do not reach family-wise statistical significance, and pair expansion (∣K∣≤2|K|\le 2) shows virtually no benefit over singleton selection (+0.40 utility points, CI [−1.59,1.98][-1.59, 1.98]) under LLM control.

Questions for authors
  1. Re-Execution and Release of Auditable Raw Trajectories for Table 2 and Thermal Runs (Clarification / Reproducibility): Since the deterministic selector runs are lightweight (Section 5.4 notes that full and simple trajectories average only 8.91 ms and 5.20 ms per trajectory), why could the 36-task deterministic selector benchmark (Table 2) and the thermal policy sweep not be cleanly re-executed prior to submission to provide complete, auditable per-task records and paired confidence intervals, rather than relying on aggregate integer-compatibility checks over lost logs? Will a complete, freshly executed trajectory log be provided during the discussion period?

  2. Sensitivity to Mis-specified Stale Priors (pap_a) and Dynamic Graph Discovery (Clarification): The conditional probing rule relies on a pre-configured stale probability pap_a (probe iff (1 - p_a)r_a > v_a). In realistic scientific workflows, how is pap_a estimated before inspecting the data, and how sensitive is CBR-VERIFY's credit efficiency when pap_a is severely mis-specified? Additionally, how does the Closure Advisor handle workflows where an LLM agent dynamically writes new analysis code or introduces new nodes/edges into G=(V,E)G=(V,E) mid-trajectory?

  3. Bridging Fixed-Contract Auditing with Open-Ended Hypothesis Scope Expansion (Suggestion for Future Work): Currently, a claim's scope scs_c and predicate ϕc\phi_c are static, and pair expansion shows limited gain over singleton seeds in the 12-task LLM cohort (+0.40 points) and coincides with singleton means on the main thermal curves (Figure 5). Could the Closure Advisor be extended to propose scope-expanding actions (e.g., suggesting which additional runs, split regimes, or boundary conditions to test next to make a scientific hypothesis more broadly solid), and what structural properties of a workflow determine when pair expansion (k=2k=2) meaningfully outperforms singleton closure (k=1k=1)?

Reviewer_SYakOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper is trying to solve the following problem: how should an AI agent decide under a budgeted situation what experiments to redo when some experimental conditions are changed, what conclusions are still valid, and which claims are supported under the current budget. The paper formalizes this problem and offers an algorithm to solve this problem by constructing first a DAG for the computations (since some of the claims may share computations), and casts is into an optimization problem to maximize the claims weighted by importance that can be made under a certain budget. They claim to achieve better performance with lower cost under various tasks compared to existing methods.

Critical strengths
  1. The paper addresses an important problem in scientific agent workflows: how to efficiently update and verify scientific conclusions when underlying data, assumptions, or evaluation protocols change.
  2. The novelty of this approach is taking into consideration that scientific claims have shared evidence which could potentially change the budget spending landscape when the agent tries to make decisions.
Critical weaknesses
  1. The algorithm is very similar to the cited work Epigraph, so I think the algorithm itself lacks novelty.
  2. The paper is missing some baselines: e.g. exact knap-sack solver, random scheduling method, or adaptation on Epigraph method.
  3. Scientific Revision tasks contain 5 scientific claims and 3 revision rounds, even with brute-force method the search space is pretty small, so the empirical results are not strong enough to prove the proposed method is effective in a real-world scenario.
Questions for authors
  1. line 255: “Original records for Table 2 have not been recovered for independent reaggregation; author-checked corrections now pass the aggregate consistency checks in Appendix E.1.” Why were the experiments not rerun but relied on author-corrected checks?
  2. How are the results compared to exact knap-sack and Epigraph?
Reviewer_gU6iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CBR-VERIFY, a framework for maintaining and revising scientific evidence in agent-based workflows under computational budgets. The method represents scientific claims using executable predicates, tracks evidence validity through versioned dependency graphs, and prioritizes evidence acquisition using a claim-completion objective. Its proposed closure-selection strategy accounts for complementary prerequisites, shared acquisition costs, and the trade-off between validating archived artifacts and recomputing them. The authors evaluate the framework on synthetic scientific revision tasks and recorded thermal measurements. The reported results demonstrate improvements over several baseline strategies in evidence-supported utility and resource allocation, although the improvements depend on the experimental configuration.

Critical strengths
  1. Maintaining valid scientific conclusions as evidence and assumptions change is an important challenge for autonomous scientific agents. The distinction between evidence validity, claim evaluation, and computational resource allocation is useful.
  2. The combination of versioned evidence dependencies, executable predicates, and budget-aware scheduling provides a systematic mechanism for validating and repairing scientific workflows.
  3. The claim-completion objective captures complementary dependencies and shared acquisition costs. The analysis of supermodularity and conditional validation provides useful intuition for the proposed scheduling strategy.
  4. The experiments include deterministic scheduling comparisons, LLM-guided controllers, controlled verification-feedback comparisons, and revision tasks based on real thermal measurements.
Critical weaknesses
  1. The benefit of pairwise closure expansion is not consistently demonstrated. In Table 1, full CBR improves utility over singleton-only selection by only 0.40 percentage points. In Figure 5, full CBR and singleton selection achieve identical aggregate utilities across the reported budgets. These results leave the practical importance of pair expansion uncertain.
  2. Figure 5 shows that at a budget of seven credits per round, full CBR achieves only approximately 17% utility, compared with approximately 43% for the dependency-aware baseline. This suggests that the closure-feasibility requirement may prevent useful partial acquisitions when the budget is insufficient to complete an entire claim. Such behavior is particularly concerning for a framework designed for budget-constrained operation.
  3. The LLM comparison includes only 12 test tasks, and the reported improvements do not reach significance after the stated Holm correction.
  4. The theoretical results establish properties of the evidence contracts and restricted closure-density ranking, but do not provide an approximation guarantee for the overall budgeted selection policy. It remains unclear how the proposed heuristic compares with stronger optimization-based or closure-aware scheduling methods. Furthermore, experiments largely assume predefined dependency graphs, executable predicates, and controlled revision scenarios, leaving robustness to incomplete dependencies or imperfect scientific evidence insufficiently explored.
Questions for authors
  1. Given the negligible improvement over singleton-only planning in Table 1 and identical aggregate performance in Figure 5, under what conditions does pair expansion provide substantial benefits? Could the authors characterize these conditions and provide additional paired experiments?
  2. What explains the substantial performance degradation and resource underutilization at the seven-credit budget in Figure 5? Does the requirement that a complete closure fit within the remaining budget prevent useful partial acquisitions? How does the separately reported low-budget follow-up differ from the earlier cohort?
  3. Could the authors compare their heuristic against an exact integer-programming solution for small fixed-cost instances, or a stronger budget-aware closure-selection baseline?
  4. Can the authors provide per-task results, execution traces, and sufficient information to independently reproduce the deterministic-selector comparisons?
  5. How sensitive is performance to inaccurate staleness probabilities, incomplete dependency graphs, and different action-cost assignments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.