acceptodds
Under review as a conference paper at ICLR 2027

CREDO: Probe Signal Quality Is the Binding Constraint on Certified VLM-Pool Routing

Abstract

A per-tier false-accept bound does not bound the error of the traffic it admits; that error is set by the base rate and by how well the probe separates right answers from wrong ones. We show this on CREDO, a training-free router over a pool of vision-language models (VLMs) and, to our knowledge, the first to certify a user-set answer-quality budget at every operating point of such a pool: a finite-sample, distribution-free, per-tier bound at the user's budget α on the rate at which a tier's wrong answers are accepted, built from a rule-based probe, split-conformal thresholds and a cheapest-certified-first cascade. It covers the queries clearing those thresholds; under the default strongest-tier fallback the rest ship uncertified. On a frozen four-tier pool over six benchmarks and 9 calibration splits, the bound holds on average for the quantity it bounds and fails for the quantity a deployer reads: the conditional error of certified traffic exceeds α at 23 of 30 operating points, stable across splits. The calibration fold predicts the gap before it is measured, a label audit leaves it essentially unchanged, no weak-signal channel raises pooled discrimination appreciably on any split under the shipped pooling, and a learned fusion of the same inputs and a hidden-state probe above the registered bar still leave most operating points over budget. Routing adds no accuracy over the best fixed tier and, once the probe is charged, no cost saving over the strongest tier. The certificate's reach is real and its reach grows with the risk level, but at every level the error a deployer reads is set by the base rate and the probe rather than by the bound.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, model routing, conformal prediction, certified risk, difficulty probes, cost-accuracy trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TqmUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents CREDO, a training-free router over a pool of VLMs, and audits the practical meaning of its certificate. A rule-based probe supplies scores for wrong-stratum split-conformal calibration, followed by a fixed cascade with uncertified fallback or abstention. Experiments on six benchmarks show that controlling per-tier false-accept rates does not ensure that the conditional error of certified traffic meets the same budget. The analysis examines base rates, probe discrimination, scoring corrections, alternative probes, and pool extensions, and reports cost and accuracy comparisons with fixed tiers and adapted routing baselines.

Critical strengths
  • The paper clearly separates per-tier false-accept rates, the conditional error of certified traffic, and overall routed accuracy. It reports unfavorable results, includes the probe's inference cost, and explicitly identifies the best fixed tier selected using test labels as an oracle comparator. This transparent accounting helps readers assess the practical meaning of the certificate.
Critical weaknesses
  • The pooled exact-binomial audit does not distinguish false-accept rates conditional on fitted thresholds from validity averaged over calibration randomness, and it pools correlated outcomes from the same questions across tiers. Section 5 also claims that AUROC 0.5 implies equal true- and false-accept rates, which is not generally true. These issues weaken the statistical analysis.
  • The empirical contribution needs a clearer comparison with prior work. The CP-Router variant replaces the original prediction-set rule with CREDO's wrong-stratum calibration, limiting what that comparison establishes about the original method. COIN's cascade calibration is underspecified, while LEC's single-model VQA evaluation and general multi-model formulation receive limited discussion. These gaps weaken the positioning of the audit.
Questions for authors
  • What randomness does the probability in Eq. 1 average over, and how do the pooled exact-binomial tests account for shared calibration thresholds and correlated predictions across tiers?
  • Under what assumptions does AUROC 0.5 imply equal true- and false-accept rates in Section 5? Which conclusions remain supported when the analysis uses acceptance rates at the selected thresholds rather than AUROC?
  • Which empirical findings about VLM routing go beyond prior work on calibration, routing, and the conditional error of accepted answers? Given the learned fusion and hidden-state experiments, which conclusions extend beyond the rule-based probe, and which depend on its scoring rule or inference cost?
  • Section 2 attributes to CP-Router the premise that a per-model conformal threshold makes a routed answer trustworthy at the stated budget. Which guarantee in the original method supports that attribution, and which risk quantity does it control?
  • What risk quantity and confidence level does the COIN implementation control, how are its thresholds selected, and how is the guarantee preserved when tier-specific thresholds are composed into the cascade?
  • How do the learned fusion and hidden-state probes change the oracle coverage curves in Appendix D and the risk–coverage trade-off of directly controlling the conditional error of accepted answers?
  • Appendix H restricts analysis to agreeing rows after missing the 0.97 gate, retaining only 49% of MathVista for the 7B probe. Is this fallback specified in advance, and how sensitive is the conclusion about probe capacity to excluding the remaining questions?
1 comment
AuthorsOpenReview ↗

Response to Reviewer TqmU

Thank you for the review.

What Eq. 1 averages over, and the binomial audit. The probability averages over the calibration fold and a new test query. The revision states this in Section 3 and in the scope paragraph. The pooled test treated answers as independent. On the stored records, of 45 (split, alpha) pairs, 3 are above alpha under the paper's test. A beta-binomial test that accounts for the shared thresholds places 2 above. A question-clustered bootstrap, with the lower end of its interval above alpha, places 0. The split-averaged rates are 0.047, 0.097, 0.147, 0.198 and 0.298 at alpha 0.05 to 0.30, and the 95% interval contains alpha at all five levels. The count of 23 of 30 operating points is a direct measurement of certified error and does not depend on the test.

AUROC 0.5 and TA = FA. Fixed in the revision: Section 5 now says that AUROC one half does not by itself give TA = FA at a given threshold. Eq. 3 is an identity in TA, FA and the base error rate pi, and applies at the selected threshold. The conclusions that use acceptance rates at that threshold are unchanged. These are the 23-of-30 count and the calibration-fold replay, which predicts test-fold certified error with Pearson r 0.90. The association between cell AUROC and the gap fraction, Spearman rho -0.94, is reported as an empirical result.

Findings beyond prior work. The paper measures, on a VLM pool, a certificate whose per-tier quantity is bounded and whose deployer-facing quantity exceeds alpha at 23 of 30 points. It predicts that gap from the calibration fold. Two conclusions hold beyond the rule probe.

  • Learned fusion raises pooled AUROC from 0.561 to 0.679 (permuted labels 0.494). The certificate built on it is over budget at 21 to 23 of 30 points.
  • The hidden-state probe raises pooled AUROC from 0.563 to 0.718. On the matched halved fold, the count moves from 24 to 20 of 30 (20 to 25 across splits).

The cost statements refer to this serving stack.

CP-Router attribution. CP-Router routes by prediction-set size. Fixed in the revision: Section 2 no longer attributes the premise to CP-Router and RACER. It words the premise as what a per-model conformal threshold invites a deployer to read, and the audit tests that reading. The original rule needs option-level probabilities, and the stored response matrix does not hold them. The table labels the CP-Router row a variant.

COIN. The implementation targets per-tier selective error, P(wrong | accepted), using an exact Clopper-Pearson upper limit at delta/G, where delta is 0.05 and G is 20. Each tier tests 20 empirical thresholds and keeps the qualifying threshold with the highest coverage. On the stored records, on the cascade slice, 0 of 6, 6 of 15, 10 of 19 and 6 of 40 nonempty cells are above alpha at alpha 0.10, 0.15, 0.20 and 0.30. On the whole accepted region the counts are 0 of 6, 2 of 15, 1 of 19 and 1 of 52. Per-tier calibration gives no guarantee for the subset that a later tier receives.

LEC. The revision adds a same-pool comparison with LEC's two-model rule, on the rule-probe scores and the same nine splits (Section 4.2; Table 20 in Appendix N; the numbers are in our reply to Reviewer DunZ).

Oracle coverage with stronger probes. At alpha 0.10, median oracle coverage across 24 cells, averaged over nine splits, is 0.009 for the rule probe. Pooled and per-benchmark logistic fusion reach 0.106 and 0.091. Boosted trees reach 0.092 and 0.053. The last-token hidden-state probe reaches 0.168. Test labels select these thresholds, so the values show the room each score offers. Appendix N has the oracle risk-coverage values of every score family at five budgets (Table 22) and the measured coverage and risk of the hidden-state cascade after per-tier calibration (Table 21).

Appendix H fallback. The fallback was registered before the first row that contained 7B outputs was committed, and the registered text has not changed since (results/rebuttal_20261106/d9_fallback_registration.json in the supplement records this order). On the stored records, the pooled AUROC of 7B minus 3B is -0.0091 on the same test fold when the regenerated channels are available on agreeing rows, -0.0010 on all rows with frozen labels, and -0.0025 on all rows with the 7B labels rescored. All three give the registered reading "not capacity-limited" (a difference of at most +0.05).

Reviewer_CtpHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose CREDO, a training-free router/cascade over a frozen VLM pool. A cheap rule-based probe scores each tier, and a per-tier split-conformal threshold on the wrong stratum certifies a finite-sample, distribution-free bound on P(accept∣wrong)≤α. The paper’s main contribution is an audit: the certified quantity and the deployer-facing quantity P(wrong∣accept) differ, and the latter exceeds α at 23 of 30 operating points. The authors explain this gap via a base-rate identity (Eq. 3), and test stronger probes, direct accepted-region certification, and a fifth tier.

Critical strengths
  1. Rigorous audit. Nine split seeds, in-pool baselines, per-point realized risk, a three-stage label wash, pre-registered extensions, and violations reported rather than patched.
  2. Strong reproducibility. Frozen response matrix, CPU-reproducible pipeline, macro-generated numerals, and released artifacts.
  3. Useful negative result. The distinction between false-accept bound and selective error is clearly demonstrated and mechanistically explained.
Critical weaknesses
  1. Limited methodological novelty. CREDO is a rule probe + wrong-stratum conformal thresholds + cascade fallback. The probe is weak (pooled AUROC 0.563), and the method shows no cost or accuracy advantage. The central gap is essentially a denominator identity (Eq. 3), not a new theoretical contribution; the paper’s value is measurement, not method.
  2. Narrow empirical scope. One frozen four-tier pool, six English VQA benchmarks, one serving stack. The nine splits resample the same frozen matrix and are not independent; the authors acknowledge these are not confidence intervals. The quantitative 23/30 result does not directly generalize.
  3. Bad presentation. For example, the conclusion paragraph in the introduction consists entirely of noun phrases.
Questions for authors
  1. Can the qualitative Eq. 3 prediction be validated on at least one independent dataset or model pool? The current nine splits are not independent samples.
  2. Can the authors provide a fairer same-construction comparison with COIN and CP-Router, rather than only disclosed variants?
  3. How much does the tier-1 cost constant (3.07 vs. clean 2.04 s/item) affect the “no cost advantage” conclusion? Can the main table be rerun with the clean constant?
1 comment
AuthorsOpenReview ↗

Response to Reviewer CtpH

Thank you for the review.

Contribution. CREDO combines a training-free rule probe, wrong-stratum split-conformal thresholds and a cascade that takes the first certified tier in a fixed capacity order. To our knowledge, it is the first certificate of a user-set false-accept budget over a VLM pool. The audit then predicts the gap before it measures it. Replaying the cascade on the calibration fold predicts the test-fold certified error with Pearson r 0.90 over 30 operating points. The base rate alone gives 0.85. The replay matches the test-fold verdict (above or below alpha) at 27 of 30 points, and at 26 to 30 across splits. Over 120 (tier, benchmark, alpha) cells, r is 0.69. Cell AUROC and the gap fraction have Spearman rho -0.94. Eq. 3 supplies the mechanism, and these measurements test it.

Presentation. Fixed in the revision: the contribution paragraph is now written in full sentences.

Independent dataset or model pool. Two parts of the paper go beyond the main pool.

  • Appendix I: on the design-external AI2D benchmark (3,088 questions), transferred thresholds meet the selective-error target at 4 of 5 budgets. Calibrating on AI2D's own split gives coverage 18.7 points above the design mean at alpha 0.10.
  • Appendix K: a fifth tier from a third model family leaves the count of over-budget operating points at 23 of 30 on the published split, the same as the four-tier pool.

Across the nine splits, 23 to 25 of 30 operating points are over budget.

A same-construction comparison with COIN and CP-Router. The paper runs COIN through the same cascade as CREDO. On the stored records, we also ran a per-tier COIN implementation that targets selective error with an exact Clopper-Pearson upper limit at delta/G, with delta 0.05 and G 20. At alpha 0.10, its median coverage is 0.039 at risk 0.066 (over the six splits where it accepts answers) and cost 4.08 s/item, against 0.281, 0.225 and 3.86 for CREDO. At alpha 0.20 the values are 0.157, 0.162 and 4.02, against 0.471, 0.261 and 3.70. On the cascade slice, 0 of 6, 6 of 15, 10 of 19 and 6 of 40 nonempty cells are above alpha at alpha 0.10, 0.15, 0.20 and 0.30. On the whole accepted region the counts are 0 of 6, 2 of 15, 1 of 19 and 1 of 52. Per-tier calibration gives no guarantee for the subset that a later tier receives. We added LEC's two-model rule on the same pool and scores. Its values are 0.141, 0.124 and 4.02 at alpha 0.10, and 0.363, 0.201 and 3.90 at alpha 0.20. At these two budgets the three rules trade risk against coverage on the same scores; Table 20 gives all five budgets. The original CP-Router rule needs option-level probabilities, and the stored response matrix does not hold them. The table therefore labels that row a variant.

The tier-1 cost constant. On the stored records, we recomputed the main table at the clean constant of 2.04 s/item. CREDO at alpha 0.10 moves from 3.75 to 2.72 s/item, and its cheapest pooled point is 2.50, against 1.26 for the strongest tier alone. No pooled point is below the strongest tier at either constant, so the paper's cost conclusion does not depend on the constant. Quality and risk do not change in any row. The cost of the probe is a property of the serving stack. A hidden-state probe needs one prefill of 0.3207 GPU-seconds per query.

Reviewer_DunZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CREDO which uses probe scores and conformal thresholds to route queries through a pool of VLMs. The guarantee limits the probability that an incorrect answer is accepted. The experiments show that across four models and six benchmarks accepted-answer error exceeds the nominal budget at 23 of 30 operating points. The authors also run several other experiments around signal ablations, test stronger probes, audit the scoring, and check additional models and benchmarks.

Critical strengths

The paper studies a practical concern around statistical certificates' potential to be mistaken for a guarantee about the error of answers users actually receive, even when it controls a different quantity. Measuring this mismatch in VLM routing is useful. The authors also clearly distinguish certified traffic from fallback responses.

The authors do examine the negative results carefully. They also run several additional experiments to audit the scoring, test stronger probes, account for probe costs, and check sensitivity across nine partitions.

The paper also has a strong discussion of the limitations, with the authors acknowledging that the partitions reuse the same response matrix and are not independent replications.

Critical weaknesses

I believe the novelty of this paper is mainly empirical. The calibration approach and the distinction between the two error rates are already established, as the authors acknowledge. LEC already includes VQA selective prediction and a general formulation for routing across multiple models, although its routing experiments are text-only. So the added contribution here is this specific audit of a VLM pool. I am not convinced that the actual guarantee supports the broader framing of certifying an answer-quality budget.

The experimental setup also limits what we can conclude from the negative results. The probe alone costs 3.07 GPU-seconds per query, while the strongest tier costs 1.26. So there cannot be cost savings with this implementation. CP-Router's defining prediction-set rule is replaced, and several other baselines use simplified substitutes. The authors disclose this, but it still weakens conclusions about the existing methods. More importantly, they do not combine the stronger probe with a method that controls the error rate among accepted answers at the system level. That comparison would help separate limitations of the available signals from limitations of the guarantee being used.

Questions for authors

What is the advance over LEC, and can you evaluate LEC on the same model pool and scores?

What happens to error rates, coverage, and cost if the stronger hidden-state probe is combined with system-level control of errors among accepted answers? Please include the cost of all inference this setup requires. The COIN and oracle-coverage analyses seem to use only the original probe.

Does the stated guarantee average over both calibration samples and future queries? How does the binomial audit in Section 4.3 handle shared calibration thresholds and repeated questions across tiers? If the guarantee averages over calibration samples, why is the fixed-split result in Appendix K treated as a violation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer DunZ

Thank you for the review.

Advance over LEC and a same-pool evaluation. CREDO is, to our knowledge, the first training-free router to certify a user-set false-accept budget at every operating point of a VLM pool. LEC's routing experiments are text-only. Fixed in the revision: the text now states LEC's selective-prediction scope, including VQA. On the stored records, we ran LEC's two-model rule on the same pool, with the rule-probe scores and the same nine splits. The pair 3B and 32B was fixed before the run. At alpha 0.10, medians across splits:

  • Coverage: 0.281 for CREDO, 0.141 for the LEC pair, 0.039 for COIN.
  • Risk among accepted answers: 0.225, 0.124 and 0.066 (COIN over the six splits where it accepts answers).
  • Cost: 3.86 s/item for CREDO, 4.02 for the LEC pair, 4.08 for COIN.

At alpha 0.20, coverage is 0.471 for CREDO, 0.363 for the LEC pair and 0.157 for COIN, at risk 0.261, 0.201 and 0.162. Cost is 3.70, 3.90 and 4.02 s/item. At these two budgets the three rules trade risk against coverage on the same scores. CREDO accepts answers in all 54 benchmark-split cells at alpha 0.10. The LEC pair accepts none in 19 of the 54 cells, and COIN in 48; at alpha 0.20 these counts are 0 and 44. Table 20 in Appendix N gives all five budgets. These runs apply LEC's calibration and routing rules to CREDO's scores. LEC's own uncertainty estimator is outside the comparison.

What the guarantee covers. The bound is P(accept | wrong) <= alpha for each tier. Fixed in the revision: the abstract and Section 2 now name the certified quantity as a false-accept budget. The paper measures the quantity a deployer reads, the error among certified answers. That error exceeds alpha at 23 of 30 operating points.

Baselines and cost. The paper discloses the substitutions in each adapted baseline (daggers in Table 1; Appendix C). The CP-Router variant runs on the same test folds as CREDO. The probe cost is a property of this serving stack. A hidden-state probe needs one prefill of 0.3207 GPU-seconds per query.

Hidden-state probe with per-tier COIN calibration and a measured cascade. On the stored records, the last-token hidden-state probe reaches pooled AUROC 0.718 on the published split, against 0.563 for the rule probe. With test labels choosing the thresholds (Table 22), its oracle coverage at alpha 0.10 is 0.168, against 0.009 for the rule probe; at alpha 0.15, 0.20 and 0.30 it is 0.271, 0.425 and 0.818. Learned fusion of the original signals gives 0.106 (pooled logistic) and 0.091 (per-benchmark logistic) at alpha 0.10. These values show the room each score offers. The experiment that ran calibrates COIN per tier on the hidden-state scores and then measures the cascade (Table 21). The final accepted set is not calibrated jointly. The splits that certify answers at alpha 0.05, 0.10, 0.15, 0.20 and 0.30 number 0, 3, 8, 9 and 9. Of the certifying benchmark points, 0 of 3, 1 of 16, 2 of 31 and 5 of 47 are above the target at alpha 0.10 to 0.30. At alpha 0.10, selective risk on the three certifying splits has median 0.068 (0.039 to 0.089), the largest one-sided 95% Clopper-Pearson upper limit is 0.117, coverage over all nine splits has median 0 (range 0 to 0.121), and mean cost with fallback is 1.12 to 1.46 s/item. The upper limit is above the target on 1 of 3, 1 of 8, 1 of 9 and 0 of 9 certifying splits. For the cost of all inference, each query pays the 0.3207 GPU-second prefill plus the full measured rate of the answering tier. Appendix N reports the full read-out at all five budgets (Tables 21 and 22). Appendices L and M report the hidden-state probe and the learned fusion.

Eq. 1 and the binomial audit. The probability in Eq. 1 averages over the calibration fold and a new test query. The revision states this in Section 3 and in the scope paragraph. The pooled binomial test treated answers as independent. On the stored records, of 45 (split, alpha) pairs, 3 are above alpha under the paper's test. A beta-binomial test that accounts for the shared thresholds places 2 above. Under a question-clustered bootstrap, no pair has a 95% interval whose lower end is above alpha. The split-averaged rates are 0.047, 0.097, 0.147, 0.198 and 0.298 at alpha 0.05 to 0.30. The 95% interval contains alpha at all five levels. Fixed in the revision: Appendix K now describes its result as an excess on one fixed split, which the marginal bound allows. The count of 23 of 30 is a direct measurement of certified error and does not depend on this test.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Two numbers from our replies, as the revised PDF defines them

We traced every number in the paper to its artifact and uploaded a revision. Two numbers in our earlier replies need their exact definition, which the revised PDF now states.

Appendix H fallback (reply to TqmU). The value -0.0091 is the 7B-minus-3B pooled AUROC on the full test fold, with the regenerated 7B channels left out on questions where the stored and regenerated answers disagree. The revised Appendix H adds the comparison on the 2,777 test questions where the answers agree, with 3B evaluated on the same questions: the pooled logprob/vote AUROC with the rule is 0.557 for 7B and 0.558 for 3B. Every reading gives the registered result "not capacity-limited" (a difference of at most +0.05).

Per-tier COIN at alpha 0.10 (reply to CtpH). The risk of 0.066 is the median over the six splits where COIN accepts answers. The coverage of 0.039 is the median over all nine splits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.