- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Tool-using vision-language agents choose their next tool from what they have already seen. The paper names a failure mode, policy-recomposition failure: a false-certification guarantee calibrated for one evidence-gathering procedure may break when the agent changes its tool order, routing or stopping rule. CETRA instead calibrates small reusable pieces once, per action and per cell of the history. Each tool output becomes a betting increment, the increments are multiplied into a certificate, and the claim is certified when the product crosses a threshold. Theorem 1 bounds false certification for every admissible policy under a mean-domination condition (Eq. 6). Experiments use a simulator plus ChartQA and DocVQA with a frozen Qwen3-VL-8B and a suite of deployment policies. CETRA keeps worst-policy risk within its bound on both real datasets with one calibration per dataset. It clearly beats static certification but loses coverage to per-policy recalibration.
Critical strengths
- The target is clear and useful. Certifying a prespecified assertion, given that it is unsupported, is cleanly separated from selective risk among asserted claims.
- The paper is unusually honest about scope. It says the sequential closure is standard and that its empirical check does not verify the key assumption.
- Calibration failure is charged explicitly to the error budget, and cells without calibration support get neutral increments. Both are careful, conservative choices.
- The validity criterion is strict: the worst policy under a one-sided Clopper-Pearson bound, not an average risk. The negative baselines fail clearly, so policy-recomposition failure is shown convincingly.
Critical weaknesses
-
The real-data scorer is deterministic given the input, history and action, with cached tools and greedy decoding. Eq. 6 then becomes a pointwise envelope: every null score is at most Γ times the cell mean. When calibration succeeds, the cell mean is at most its upper bound. So every null increment is at most one, the certificate never exceeds one, and no unsupported assertion can be certified. Theorem 1 thus predicts real-data risk no higher than the calibration-failure budget. The reported worst-policy risks are several times that budget and close to the sequential budget. Calibration failure alone is very unlikely to explain this, so the declared Γ must be exceeded on many null histories.
-
The admissible class Π is defined as the policies that satisfy Eq. 6. "Validity across admissible policies" then restates the assumption. The theory never says whether the fixed orders missing from the logs are admissible. The sufficient envelope in Eq. 9 takes a supremum over histories that contain the input and the assertion. With a deterministic scorer, it becomes the worst null instance's score over the cell mean. Policy independence is then bought by worst-case domination, not by structure that supports reuse. Such a factor puts the threshold above every null score in the cell, so coverage may be small for a realistic scorer.
-
The PDF ends with the references. The main text relies on Appendices A to D, on Tables 13, 14, 20 and 21, and on a supplement. These hold the formal assumption list, all proofs, Theorem 2, the joint-control result, tool and policy details, per-policy results, ablations, stress tests and the natural-error breakdown.
-
Figure 1 contradicts the method. Its increment formula differs from Eq. 7, and its fold mixture adds an averaging factor that Eq. 8 lacks. It states the threshold and bound with the overall target instead of the sequential budget. It calls the certificate an anytime-valid e-process, which Section 5 disclaims. It lists policies that visit unsupported cells as out of scope, although the text covers them through neutral increments. It shows a chest radiograph, unrelated to charts or documents, and its text is tiny at print size. It should be redrawn to match the equations.
Questions for authors
- In the real-data runs, is the score a deterministic function of the input, assertion, history and action? If so, do you agree that under Eq. 6 and successful calibration no null can be certified? How do you reconcile this with the observed risk?
- What Γ values were declared, and on which data and at what stage were they chosen? Did any test result inform them? How often do null scores exceed Γ times the bound?
- Is the shift-factor estimator part of CETRA as evaluated, and what does it guarantee? What does the "remove sh
Author Response to Reviewer 1tJy
Response to Rev.1tJy
We thank you for a precise review and answer your questions in order.
Is the score deterministic, and can a null be certified under Equation 6? Yes on both counts, and we agree with the consequence you draw. Decoding is greedy and every tool, the scorer, the history encoder and the cell assignment are deterministic at inference. Under Equation 6 at the declared factor and on a successful calibration event, no null trajectory can cross the threshold, and Theorem 1 predicts risk at most 0.02. To reconcile this with the observed 0.071 and 0.068 we counted false crossings per partition. Each of the twenty partitions has its own calibration artifact, and every evaluated policy records false certifications in all twenty, so attributing them to calibration failure would require all twenty artifacts to fail, which has probability at most 0.02. We therefore reach your conclusion: the declared factor is exceeded on visited null histories. We no longer offer calibration failure as a resolution.
What factor values were declared, and when were they chosen? The declared envelope factor was fixed at 1.5 on the score-training split, before any test-partition evaluation, and held constant across both datasets, all seven deployment policies and all twenty partitions. No test-partition risk, coverage or certification outcome informed it, and the sweep in the sensitivity table is a post-hoc analysis rather than the selection procedure. Appendix C.1 now states this, since "without test labels" did not exclude selection on test outcomes.
How often do null scores exceed the declared factor times the bound? On ChartQA, 1,646 of 38,000 visited non-neutral null invocations, or 4.33%. The new appendix gives the breakdown you asked for. By cell support, splitting at 200 contributing sources gives 5.27% below and 2.89% above, a ratio of 1.8. By action, the rates are 3.47% for crop, 5.10% for OCR, 3.47% for table and 5.12% for calculate. The exceedances are therefore not concentrated in a sparse margin or a single tool.
Is the shift-factor estimator part of CETRA as evaluated, and what does it guarantee? It is not part of CETRA as evaluated, it is used in no reported experiment, and it guarantees nothing. It is a bootstrap percentile summary of a within-cell regression quantile, which is neither an upper confidence bound on the sufficient envelope nor a certified bound of any kind, as Appendix B.8 states. A certified factor would require the simultaneous coverage condition of Theorem 2, which we do not supply. The Reproducibility Statement's description of it as an estimator "and its guarantee" was wrong and has been corrected, as has the claim in Section 2 that the enabling condition is "estimable".
The ablation. Your final question is truncated in the version available to us, so we answer what we believe it asks. The "no shift correction" ablation sets the declared factor to 1 for every tuple, so the increment denominator reduces to the calibration bound alone. It is a setting of the declared factor rather than the removal of a separate mechanism, which is why its ChartQA row coincides with the corresponding row of the sensitivity sweep; the appendix now says so. We recognise that our terminology invited the confusion, since "shift" named the declared factor, the calibration budgets, this ablation and the unrelated tool-kernel perturbation. The revision reserves "shift" for the stress test and calls the declared quantity the envelope factor. If your question concerned something else, we will answer it directly.
Admissibility restating the assumption. We agree that Theorem 1 is conditional, and that defining the admissible class by the condition makes the uniformity statement close to definitional. What is not definitional is that the sufficient envelope is a property of a cell and a history domain rather than of a policy, which is what would make one calibration object reusable across policies if the cells were adequate. Whether a policy stays inside supported domains is checkable from logs; whether the envelope holds there is not, and our audit shows that it does not at the declared factor. We also agree that with a deterministic scorer the sufficient envelope reduces to the worst null score in the cell divided by the cell mean, and that a factor large enough to dominate it would place the threshold above every null score and leave little coverage. We now present this as the central open problem rather than as a solved one, and the revision no longer claims that the characterisation is operational.
Structure and Figure 1. The appendices are now in the main PDF after the references. Figure 1 is redrawn to match Equations 7, 8 and 11 and the stated bound; the e-process description is removed, unsupported cells are shown in scope with neutral increments, the radiograph is replaced by chart and document icons, and policies retrained after calibration are listed as out of scope.