acceptodds
Under review as a conference paper at ICLR 2027

CETRA: Composable Evidence Certificates for Tool-Adaptive Risk Control of Multimodal Assertions

Abstract

Adaptive multimodal agents select tools using earlier observations, so calibration for one acquisition policy need not survive policy recomposition. We introduce CETRA, which converts tool outputs into action–history evidence increments for assertions fixed before acquisition. Cross-fitted calibration bounds logging-reference null means, while unsupported configurations contribute neutral increments. Under independent source-level calibration, stable tools, and explicit full-history conditional null-mean bounds, the product is a nonnegative supermartingale for each successful calibration artifact. Accounting for calibration failure yields anytime false-certification control across admissible acquisition and stopping policies. We evaluate CETRA in CETRA-Sim, ChartQA, and DocVQA. At , CETRA passes the policy-wise empirical confidence-bound checks on both real datasets, with worst-policy risk estimates and , while reusing one calibration object per dataset across seven policies. Coverage exceeds static certification by and percentage points. Policy-specific recalibration achieves higher coverage with separate full allocations, whereas CETRA achieves higher coverage under the evaluated matched total calibration-trajectory budgets. These results characterize the empirical coverage–reuse trade-off while keeping the assumptions required for theoretical certification explicit.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Adaptive multimodal agents, Tool-augmented reasoning, Conformal risk control, Anytime-valid inference, Policy recomposition, Reliable AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1tJyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Tool-using vision-language agents choose their next tool from what they have already seen. The paper names a failure mode, policy-recomposition failure: a false-certification guarantee calibrated for one evidence-gathering procedure may break when the agent changes its tool order, routing or stopping rule. CETRA instead calibrates small reusable pieces once, per action and per cell of the history. Each tool output becomes a betting increment, the increments are multiplied into a certificate, and the claim is certified when the product crosses a threshold. Theorem 1 bounds false certification for every admissible policy under a mean-domination condition (Eq. 6). Experiments use a simulator plus ChartQA and DocVQA with a frozen Qwen3-VL-8B and a suite of deployment policies. CETRA keeps worst-policy risk within its bound on both real datasets with one calibration per dataset. It clearly beats static certification but loses coverage to per-policy recalibration.

Critical strengths
  • The target is clear and useful. Certifying a prespecified assertion, given that it is unsupported, is cleanly separated from selective risk among asserted claims.
  • The paper is unusually honest about scope. It says the sequential closure is standard and that its empirical check does not verify the key assumption.
  • Calibration failure is charged explicitly to the error budget, and cells without calibration support get neutral increments. Both are careful, conservative choices.
  • The validity criterion is strict: the worst policy under a one-sided Clopper-Pearson bound, not an average risk. The negative baselines fail clearly, so policy-recomposition failure is shown convincingly.
Critical weaknesses
  • The real-data scorer is deterministic given the input, history and action, with cached tools and greedy decoding. Eq. 6 then becomes a pointwise envelope: every null score is at most Γ times the cell mean. When calibration succeeds, the cell mean is at most its upper bound. So every null increment is at most one, the certificate never exceeds one, and no unsupported assertion can be certified. Theorem 1 thus predicts real-data risk no higher than the calibration-failure budget. The reported worst-policy risks are several times that budget and close to the sequential budget. Calibration failure alone is very unlikely to explain this, so the declared Γ must be exceeded on many null histories.

  • The admissible class Π is defined as the policies that satisfy Eq. 6. "Validity across admissible policies" then restates the assumption. The theory never says whether the fixed orders missing from the logs are admissible. The sufficient envelope in Eq. 9 takes a supremum over histories that contain the input and the assertion. With a deterministic scorer, it becomes the worst null instance's score over the cell mean. Policy independence is then bought by worst-case domination, not by structure that supports reuse. Such a factor puts the threshold above every null score in the cell, so coverage may be small for a realistic scorer.

  • The PDF ends with the references. The main text relies on Appendices A to D, on Tables 13, 14, 20 and 21, and on a supplement. These hold the formal assumption list, all proofs, Theorem 2, the joint-control result, tool and policy details, per-policy results, ablations, stress tests and the natural-error breakdown.

  • Figure 1 contradicts the method. Its increment formula differs from Eq. 7, and its fold mixture adds an averaging factor that Eq. 8 lacks. It states the threshold and bound with the overall target instead of the sequential budget. It calls the certificate an anytime-valid e-process, which Section 5 disclaims. It lists policies that visit unsupported cells as out of scope, although the text covers them through neutral increments. It shows a chest radiograph, unrelated to charts or documents, and its text is tiny at print size. It should be redrawn to match the equations.

Questions for authors
  • In the real-data runs, is the score a deterministic function of the input, assertion, history and action? If so, do you agree that under Eq. 6 and successful calibration no null can be certified? How do you reconcile this with the observed risk?
  • What Γ values were declared, and on which data and at what stage were they chosen? Did any test result inform them? How often do null scores exceed Γ times the bound?
  • Is the shift-factor estimator part of CETRA as evaluated, and what does it guarantee? What does the "remove sh
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 1tJy

Response to Rev.1tJy

We thank you for a precise review and answer your questions in order.

Is the score deterministic, and can a null be certified under Equation 6? Yes on both counts, and we agree with the consequence you draw. Decoding is greedy and every tool, the scorer, the history encoder and the cell assignment are deterministic at inference. Under Equation 6 at the declared factor and on a successful calibration event, no null trajectory can cross the threshold, and Theorem 1 predicts risk at most 0.02. To reconcile this with the observed 0.071 and 0.068 we counted false crossings per partition. Each of the twenty partitions has its own calibration artifact, and every evaluated policy records false certifications in all twenty, so attributing them to calibration failure would require all twenty artifacts to fail, which has probability at most 0.02. We therefore reach your conclusion: the declared factor is exceeded on visited null histories. We no longer offer calibration failure as a resolution.

What factor values were declared, and when were they chosen? The declared envelope factor was fixed at 1.5 on the score-training split, before any test-partition evaluation, and held constant across both datasets, all seven deployment policies and all twenty partitions. No test-partition risk, coverage or certification outcome informed it, and the sweep in the sensitivity table is a post-hoc analysis rather than the selection procedure. Appendix C.1 now states this, since "without test labels" did not exclude selection on test outcomes.

How often do null scores exceed the declared factor times the bound? On ChartQA, 1,646 of 38,000 visited non-neutral null invocations, or 4.33%. The new appendix gives the breakdown you asked for. By cell support, splitting at 200 contributing sources gives 5.27% below and 2.89% above, a ratio of 1.8. By action, the rates are 3.47% for crop, 5.10% for OCR, 3.47% for table and 5.12% for calculate. The exceedances are therefore not concentrated in a sparse margin or a single tool.

Is the shift-factor estimator part of CETRA as evaluated, and what does it guarantee? It is not part of CETRA as evaluated, it is used in no reported experiment, and it guarantees nothing. It is a bootstrap percentile summary of a within-cell regression quantile, which is neither an upper confidence bound on the sufficient envelope nor a certified bound of any kind, as Appendix B.8 states. A certified factor would require the simultaneous coverage condition of Theorem 2, which we do not supply. The Reproducibility Statement's description of it as an estimator "and its guarantee" was wrong and has been corrected, as has the claim in Section 2 that the enabling condition is "estimable".

The ablation. Your final question is truncated in the version available to us, so we answer what we believe it asks. The "no shift correction" ablation sets the declared factor to 1 for every tuple, so the increment denominator reduces to the calibration bound alone. It is a setting of the declared factor rather than the removal of a separate mechanism, which is why its ChartQA row coincides with the corresponding row of the sensitivity sweep; the appendix now says so. We recognise that our terminology invited the confusion, since "shift" named the declared factor, the calibration budgets, this ablation and the unrelated tool-kernel perturbation. The revision reserves "shift" for the stress test and calls the declared quantity the envelope factor. If your question concerned something else, we will answer it directly.

Admissibility restating the assumption. We agree that Theorem 1 is conditional, and that defining the admissible class by the condition makes the uniformity statement close to definitional. What is not definitional is that the sufficient envelope is a property of a cell and a history domain rather than of a policy, which is what would make one calibration object reusable across policies if the cells were adequate. Whether a policy stays inside supported domains is checkable from logs; whether the envelope holds there is not, and our audit shows that it does not at the declared factor. We also agree that with a deterministic scorer the sufficient envelope reduces to the worst null score in the cell divided by the cell mean, and that a factor large enough to dominate it would place the threshold above every null score and leave little coverage. We now present this as the central open problem rather than as a solved one, and the revision no longer claims that the characterisation is operational.

Structure and Figure 1. The appendices are now in the main PDF after the references. Figure 1 is redrawn to match Equations 7, 8 and 11 and the stated bound; the e-process description is removed, unsupported cells are shown in scope with neutral increments, the radiograph is replaced by chart and document icons, and policies retrained after calibration are listed as out of scope.

Reviewer_hGifOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CETRA certifies prespecified assertions under adaptive multimodal tool acquisition. It calibrates action–history support scores via source-level cross-fitting and accumulates normalized evidence increments until certification or termination. Under a full-history conditional mean-domination assumption (A5), it yields an anytime false-certification bound that accounts for calibration uncertainty. Experiments on CETRA-Sim, ChartQA, and DocVQA study policy recomposition, calibration reuse, and coverage–cost trade-offs.

Critical strengths
  1. Meaningful reliability problem: changing tool order, routing, or stopping can invalidate a calibrated procedure, and calibrating reusable action–history evidence rather than each full policy is an interesting response.
  2. Comprehensive, transparent empirics: policy-level results, component ablations, budget comparisons, and sensitivity analyses, with an explicit separation of empirical risk checks from theoretical guarantees.
Critical weaknesses
  1. The theoretical guarantee’s applicability to the evaluated implementation is unresolved. A5 requires a full-history conditional null-mean bound that the calibration procedure does not establish. Appendix B.6 shows this reduces to a pointwise null-score envelope: if A5 holds and calibration succeeds, false certification is impossible, giving a calibration-failure budget of 0.02 at the nominal target 0.10. Yet the reported worst-policy risks are 0.071 and 0.068. These finite-sample summaries do not refute the theorem, but passing the nominal check does not establish theoretically certified reuse; the guarantee remains dependent on a conditional envelope whose validity is not established for the evaluated setting.
  2. The advantage of primitive-level reuse over relevant alternatives is not yet established. Policy-specific recalibration attains higher coverage with separate full allocations, while CETRA gains ~2 percentage points under matched trajectory budgets. The comparison does not evaluate simultaneous calibration of the fixed policy family under the same false-certification objective, nor separate the accounting of shared labeled sources, cached evidence, and policy-specific trajectory evaluations. Since the sequential inference machinery is standard, these comparisons would help establish the incremental value of the interface.
Questions for authors
  1. How should practitioners select the declared conditional mean-domination factors before deployment, and what supports their validity beyond empirical risk performance?
  2. Can false certifications be audited via the Appendix B.6 decomposition (frequency/location of envelope exceedances), distinguishing A5 failures from reference-mean calibration failures?
  3. How does CETRA compare with simultaneous calibration of the seven fixed policies using shared labeled sources and cached outputs, with calibration demand reported in unique labeled sources, tool invocations, and compute?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer hGif

Response to Reviewer hGif

We thank you for a reading of Appendix B.6 that was correct, and for three questions that directly shaped the revision.

Applicability of the guarantee, and auditing false certifications (W1, Q2). We followed your suggestion and audited the false certifications using that decomposition. It separates the two failure modes at different granularities: a calibration failure is a property of an entire partition's artifact and has probability at most 0.02, whereas an envelope violation is a property of an individual invocation. Every evaluated policy records false certifications in all twenty partitions, so calibration failure would require all twenty artifacts to fail, of probability at most 0.02. The violations are therefore envelope exceedances. A new appendix reports their frequency and location: 1,646 of 38,000 visited non-neutral null invocations on ChartQA, or 4.33%, at rates of 5.27% and 2.89% in the two cell-support strata and between 3.47% and 5.12% across actions. We agree that passing the nominal check does not establish certified reuse, and the abstract, Section 4.2 and the Limitations now describe the real-data results as empirical validity checks rather than as instances of the theorem.

Selecting the declared factors before deployment (Q1). For our pipeline, nothing beyond empirical risk performance supports the declared factor, and our own audit shows that it is exceeded. We state this plainly rather than presenting the appendix diagnostic as a selection method; that diagnostic carries no coverage guarantee and is used in no reported experiment. The factor was fixed at 1.5 on the score-training split before any test evaluation, which answers where it came from but not what justifies it. The only route we can identify to a justified factor is the simultaneous coverage condition of Theorem 2, which we do not supply, and Section 4.3 and the Limitations now say so.

Comparison with simultaneous calibration, and resource accounting (W2, Q3). We added the comparison you asked for: one calibrator per policy for the seven deployment policies, fitted on the same labelled sources and cached tool outputs, with a union bound over the family. On ChartQA it attains worst-policy risk 0.062 and coverage 0.442, against CETRA's 0.071 and 0.423, and both pass the empirical check. On a prespecified family of this size, simultaneous calibration is therefore preferable to primitive reuse. This is the outcome your weakness anticipated, and we report it as such; the matched-budget advantage in the original Section 7.2 reflects the stated division of trajectories rather than an intrinsic benefit, and that section has been rewritten accordingly.

The comparison does change with the size of the deployment family, and we have measured where. CETRA's calibration object is indexed by fold, action and history cell, so it does not depend on the family, whereas simultaneous calibration pays a union bound over the policy statistics. Across families of 7, 20, 50, 100 and 340 fixed policies, the largest being the enumerable action-sequence family, CETRA's worst-policy risk and coverage are flat while simultaneous coverage falls from 0.442 to 0.379. The ordering reverses between 7 and 20 policies, where the two are within 0.2 coverage points, and CETRA leads by 1.3, 3.0 and 4.2 points at 50, 100 and 340. Per-statistic support is 165 contributing sources for both methods at every family size. We report the location of this reversal as measured, and note explicitly that the union-bound radius alone predicts a later crossing and does not account for one this early; identifying the mechanism is open. We also note that at the largest family every policy is a fixed action sequence, so this result concerns large enumerable families rather than adaptive or unseen ones.

On resource accounting, making a complete-policy method valid on a policy absent from the calibration-time family required 400 new labelled sources, 350 new calibration trajectories and 1,400 new tool invocations in our setup, against none for CETRA, whose object does not depend on the family. We also evaluated such a policy directly: a router retrained after calibration. CETRA's reused object does not pass the empirical check on it, with risk 0.092 and an upper bound of 0.1038 against a target of 0.10, whereas it passes on both complete action orders that are absent from the logs. Reuse therefore survives recomposition of logged primitives but did not extend to a policy that moves the deployment distribution away from the logs. We did not determine which property of that policy accounts for the failure, and the admissibility diagnostics of Appendix A.2 do not identify it.

I thank the authors for an exceptionally honest and responsive rebuttal, but the new material resolves my two main concerns negatively rather than positively.

(W1, theory.) The authors’ audit (Appx B.9) confirms A5 is violated across the visited support (4.33% of null invocations exceed the envelope, not localized, and unfixable by support screening, tool exclusion, or larger Γ). Theorem 1 therefore applies to none of the real-data experiments. Beyond the standard (non-novel) sequential closure, the theorem’s contribution is the cell-level characterization — which the learned cells do not attain. The central claim is an open problem, not an established result.

(W2, reuse value.) The requested comparison shows simultaneous calibration is preferable to CETRA on the realistic seven-policy suite (0.062/0.442 vs 0.071/0.423, both passing). CETRA’s advantage appears only for large enumerable families (P≥20), via a mechanism the authors cannot explain, and fails for a retrained router (0.092, UCB 0.1038 > 0.10). The interface’s incremental value in the evaluated regime is not established.

Thus, I keep my score at 2.

Reviewer_ugYROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed CETRA, a framework for reusable evidence certification in multimodal agents with adaptive tool acquisition. The main motivation is that calibration guarantees established for a particular acquisition policy may not remain valid when the agent changes its tool selection, execution order, or stopping behavior. CETRA constructs action–history-level evidence increments using cross-fitted calibration and support-aware mechanisms. These increments are sequentially combined to provide anytime false-certification control under explicit assumptions, particularly a full-history conditional null-mean domination condition.

The authors evaluate CETRA on a controlled simulation benchmark, ChartQA, and DocVQA. The results show that CETRA satisfies the empirical risk checks across seven evaluated policies while improving certified coverage over static certification.

Critical strengths

This paper is well-motivated and clear presented. The paper identifies a meaningful reliability issue in adaptive multimodal agents. I particularly appreciate that the authors clearly distinguish their certification objective from related risk-control settings and explicitly state the assumptions underlying their theoretical results. The idea of constructing reusable action–history-level evidence increments, rather than calibrating complete acquisition policies, is interesting.

The paper includes both controlled simulations and real multimodal benchmarks. The comparison with policy-specific recalibration is particularly useful, as it illustrates the trade-off between calibration reuse and certified coverage. I also appreciate the matched-budget experiments, mechanism ablations, and stress tests, which help clarify the benefits and limitations of the proposed approach.

Critical weaknesses

First of all, I have to admit that I'm not an expert in this topic. So I would rate the following comments as low confidence.

My main concern is the gap between the theoretical assumptions and empirical validation. The theoretical guarantee relies on a full-history conditional mean-domination assumption, but the paper does not provide a statistically certified way to verify this condition from finite logging data. Although the experiments demonstrate successful reuse across seven policies, they do not establish whether the required condition holds for more general unseen policies. I should also acknowledge that this paper is outside my primary research area, and I found some of the theoretical arguments challenging to fully assess. Therefore, my confidence in evaluating the technical novelty and theoretical correctness is relatively limited. Nevertheless, the overall presentation appears solid, and I did not identify an obvious theoretical error.

Questions for authors

Could the authors clarify how the conditional mean-domination factor (\kappa) should be reliably estimated or validated in practice? How sensitive are the empirical results to its choice? Additionally, it would be helpful to discuss whether the proposed calibration reuse remains effective for substantially different acquisition policies or longer tool-use trajectories, particularly when the logging data provide limited support for certain histories.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer ugYR

Response to Reviewer ugYR

We thank you for engaging carefully with a topic outside your primary area, and for stating your confidence plainly. Your main concern coincides with the other reviewers', and we address it at length in the general response. In short, the paper's own data show that the conditional mean-domination assumption does not hold at the declared factor in our real-data pipeline, so the real-data results are empirical validity checks rather than certified ones. The revised main text states this rather than leaving it to an appendix.

How the domination factor should be estimated or validated, and how sensitive the results are to it. The factor was fixed at 1.5 on the score-training split before any test evaluation. We know of no procedure that certifies such a factor from finite logging data: doing so requires the simultaneous coverage condition of Theorem 2, which we do not supply, and the regression diagnostic in the appendix is explicitly not an upper confidence bound. The sensitivity is substantial and is now reported for both datasets. On ChartQA, raising the factor from 1 to 2 lowers worst-policy risk from 0.118 to 0.052 while coverage falls from 0.462 to 0.351; on DocVQA, risk falls from 0.114 to 0.031 while coverage falls from 0.506 to 0.407. Every value on that grid leaves risk above the calibration-failure budget of 0.02, which is how we know the envelope is exceeded across the whole range we tested rather than only at the value we deployed.

Whether reuse remains effective for substantially different policies, longer trajectories, and histories with limited logging support. These are the right three questions, and our answers differ across them. On substantially different policies, we now report a direct test: reuse holds for the two complete action orders that never appear in the logs, but a router retrained after calibration is not certified within the target, so reuse survives recomposition of logged behaviour and did not extend to a policy that shifts the deployment distribution. On limited support, configurations without usable calibration contribute neutral increments by construction, and making history cells finer lowers risk but raises the unsupported fraction from 0.031 to 0.142 while coverage falls from 0.449 to 0.366. On longer trajectories, our horizon is at most four actions in every experiment, and we state explicitly that longer horizons are untested; we would expect the difficulty to increase, since each additional non-neutral invocation is a further opportunity for the envelope to be exceeded.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We thank all three reviewers for unusually careful reports. Two reviewers independently identified the same central issue, and we agree with their analysis. We have revised the paper so that the main text, rather than only an appendix, states it plainly. All changes in the revised PDF are highlighted in blue.

The theorem and the real-data experiments. In our real-data pipeline every invocation score is a deterministic function of the offline artifact, the input, the assertion, the history and the action. Assumption A5 therefore specialises to a pointwise null-score envelope, and on a successful calibration event every null increment is at most one, so Theorem 1 predicts false-certification risk no larger than the calibration-failure budget of 0.02. The reported worst-policy risks are 0.071 and 0.068. We investigated whether calibration failure could account for this gap. Each of the twenty partitions carries its own calibration artifact, and every evaluated policy records false certifications in all twenty. Under A5 each such partition would have to be a calibration failure, an event of probability at most 0.02 per artifact, so the probability that all twenty fail is at most 0.02 by Markov's inequality. Assumption A5 therefore does not hold at the declared factor on part of the visited null support.

A new appendix reports an invocation-level audit. Of 38,000 visited non-neutral null invocations on ChartQA, 1,646 (4.33%) exceed the deployed envelope. The exceedances are not localised: splitting supported cells at a support of 200 gives rates of 5.27% and 2.89%, a ratio of 1.8, and the per-action rates are 3.47%, 5.10%, 3.47% and 5.12%, a ratio of 1.5 between the extremes. Neither refining the support screen nor excluding a single tool would restore the envelope. Nor would a larger declared factor: on both datasets every value of the tested grid leaves worst-policy risk above the calibration-failure budget — 0.118, 0.086, 0.071 and 0.052 on ChartQA, and 0.114, 0.092, 0.068 and 0.031 on DocVQA.

The real-data results are therefore empirical validity checks, not instances of Theorem 1, and the abstract, Section 4.2, Section 4.3 and the Limitations now say so. Theorem 1 remains a correct conditional result. Its content beyond the standard sequential closure, which we do not claim as novel, is that the enabling condition is a property of action–history cells rather than of acquisition policies. Our measurements show that the learned cells do not attain it at the declared factor, and we now present constructing cells that do as the open problem this work leaves.

New experiments. Responding to Reviewer hGif, we added the comparison with simultaneous calibration of the deployed family on shared labelled sources, together with a study of how that comparison depends on the size of the family. On the seven-policy suite, simultaneous calibration is preferable to CETRA. Its advantage reverses as the family grows. We report both results, including the one that does not favour our method, and we state explicitly that the union-bound argument does not account for where the reversal occurs.

Other revisions. The appendices are now included in the main PDF after the references; they were previously submitted only as separate supplementary material and are unchanged except where marked. We corrected two sentences that overstated what the paper establishes: Section 2 no longer describes the enabling condition as "estimable", and the Reproducibility Statement no longer refers to a shift-factor estimator "and its guarantee", since Appendix B.8 states that the diagnostic carries no coverage guarantee and it is used in no reported experiment. We also removed a terminology collision, in which the word "shift" named the declared factor, the calibration budgets, an ablation and an unrelated stress-test perturbation; the declared factor is now the "declared envelope factor" throughout, the ablation is "no envelope inflation", and "shift" is reserved for the tool-kernel stress test. Figure 1 is redrawn to match Equations 7, 8 and 11, and its caption now states that validity is conditional on Eq. (6) and that the bound is exceeded at the declared factor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.