- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents CREDO, a training-free router over a pool of VLMs, and audits the practical meaning of its certificate. A rule-based probe supplies scores for wrong-stratum split-conformal calibration, followed by a fixed cascade with uncertified fallback or abstention. Experiments on six benchmarks show that controlling per-tier false-accept rates does not ensure that the conditional error of certified traffic meets the same budget. The analysis examines base rates, probe discrimination, scoring corrections, alternative probes, and pool extensions, and reports cost and accuracy comparisons with fixed tiers and adapted routing baselines.
Critical strengths
- The paper clearly separates per-tier false-accept rates, the conditional error of certified traffic, and overall routed accuracy. It reports unfavorable results, includes the probe's inference cost, and explicitly identifies the best fixed tier selected using test labels as an oracle comparator. This transparent accounting helps readers assess the practical meaning of the certificate.
Critical weaknesses
- The pooled exact-binomial audit does not distinguish false-accept rates conditional on fitted thresholds from validity averaged over calibration randomness, and it pools correlated outcomes from the same questions across tiers. Section 5 also claims that AUROC 0.5 implies equal true- and false-accept rates, which is not generally true. These issues weaken the statistical analysis.
- The empirical contribution needs a clearer comparison with prior work. The CP-Router variant replaces the original prediction-set rule with CREDO's wrong-stratum calibration, limiting what that comparison establishes about the original method. COIN's cascade calibration is underspecified, while LEC's single-model VQA evaluation and general multi-model formulation receive limited discussion. These gaps weaken the positioning of the audit.
Questions for authors
- What randomness does the probability in Eq. 1 average over, and how do the pooled exact-binomial tests account for shared calibration thresholds and correlated predictions across tiers?
- Under what assumptions does AUROC 0.5 imply equal true- and false-accept rates in Section 5? Which conclusions remain supported when the analysis uses acceptance rates at the selected thresholds rather than AUROC?
- Which empirical findings about VLM routing go beyond prior work on calibration, routing, and the conditional error of accepted answers? Given the learned fusion and hidden-state experiments, which conclusions extend beyond the rule-based probe, and which depend on its scoring rule or inference cost?
- Section 2 attributes to CP-Router the premise that a per-model conformal threshold makes a routed answer trustworthy at the stated budget. Which guarantee in the original method supports that attribution, and which risk quantity does it control?
- What risk quantity and confidence level does the COIN implementation control, how are its thresholds selected, and how is the guarantee preserved when tier-specific thresholds are composed into the cascade?
- How do the learned fusion and hidden-state probes change the oracle coverage curves in Appendix D and the risk–coverage trade-off of directly controlling the conditional error of accepted answers?
- Appendix H restricts analysis to agreeing rows after missing the 0.97 gate, retaining only 49% of MathVista for the 7B probe. Is this fallback specified in advance, and how sensitive is the conclusion about probe capacity to excluding the remaining questions?
Response to Reviewer TqmU
Thank you for the review.
What Eq. 1 averages over, and the binomial audit. The probability averages over the calibration fold and a new test query. The revision states this in Section 3 and in the scope paragraph. The pooled test treated answers as independent. On the stored records, of 45 (split, alpha) pairs, 3 are above alpha under the paper's test. A beta-binomial test that accounts for the shared thresholds places 2 above. A question-clustered bootstrap, with the lower end of its interval above alpha, places 0. The split-averaged rates are 0.047, 0.097, 0.147, 0.198 and 0.298 at alpha 0.05 to 0.30, and the 95% interval contains alpha at all five levels. The count of 23 of 30 operating points is a direct measurement of certified error and does not depend on the test.
AUROC 0.5 and TA = FA. Fixed in the revision: Section 5 now says that AUROC one half does not by itself give TA = FA at a given threshold. Eq. 3 is an identity in TA, FA and the base error rate pi, and applies at the selected threshold. The conclusions that use acceptance rates at that threshold are unchanged. These are the 23-of-30 count and the calibration-fold replay, which predicts test-fold certified error with Pearson r 0.90. The association between cell AUROC and the gap fraction, Spearman rho -0.94, is reported as an empirical result.
Findings beyond prior work. The paper measures, on a VLM pool, a certificate whose per-tier quantity is bounded and whose deployer-facing quantity exceeds alpha at 23 of 30 points. It predicts that gap from the calibration fold. Two conclusions hold beyond the rule probe.
The cost statements refer to this serving stack.
CP-Router attribution. CP-Router routes by prediction-set size. Fixed in the revision: Section 2 no longer attributes the premise to CP-Router and RACER. It words the premise as what a per-model conformal threshold invites a deployer to read, and the audit tests that reading. The original rule needs option-level probabilities, and the stored response matrix does not hold them. The table labels the CP-Router row a variant.
COIN. The implementation targets per-tier selective error, P(wrong | accepted), using an exact Clopper-Pearson upper limit at delta/G, where delta is 0.05 and G is 20. Each tier tests 20 empirical thresholds and keeps the qualifying threshold with the highest coverage. On the stored records, on the cascade slice, 0 of 6, 6 of 15, 10 of 19 and 6 of 40 nonempty cells are above alpha at alpha 0.10, 0.15, 0.20 and 0.30. On the whole accepted region the counts are 0 of 6, 2 of 15, 1 of 19 and 1 of 52. Per-tier calibration gives no guarantee for the subset that a later tier receives.
LEC. The revision adds a same-pool comparison with LEC's two-model rule, on the rule-probe scores and the same nine splits (Section 4.2; Table 20 in Appendix N; the numbers are in our reply to Reviewer DunZ).
Oracle coverage with stronger probes. At alpha 0.10, median oracle coverage across 24 cells, averaged over nine splits, is 0.009 for the rule probe. Pooled and per-benchmark logistic fusion reach 0.106 and 0.091. Boosted trees reach 0.092 and 0.053. The last-token hidden-state probe reaches 0.168. Test labels select these thresholds, so the values show the room each score offers. Appendix N has the oracle risk-coverage values of every score family at five budgets (Table 22) and the measured coverage and risk of the hidden-state cascade after per-tier calibration (Table 21).
Appendix H fallback. The fallback was registered before the first row that contained 7B outputs was committed, and the registered text has not changed since (
results/rebuttal_20261106/d9_fallback_registration.jsonin the supplement records this order). On the stored records, the pooled AUROC of 7B minus 3B is -0.0091 on the same test fold when the regenerated channels are available on agreeing rows, -0.0010 on all rows with frozen labels, and -0.0025 on all rows with the 7B labels rescored. All three give the registered reading "not capacity-limited" (a difference of at most +0.05).