acceptodds
Withdrawn from ICLR 2027

SafeRoute: Retention-Constrained Local Evidence Control for Mixed-Shot Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language adaptation under heterogeneous support can improve average accuracy while overturning correct predictions for classes with no labeled examples. We formulate this mixed-shot setting as constrained inference over a fixed vocabulary and introduce SAFEROUTE, a score-level interface that separates the permission to change a prediction from the strength of adaptation. We prove an exact observable characterization of universal pointwise retention: without the query label, every unsupported frozen winner must be preserved, while supported frozen winners define the maximal open route on which top-1 changes are admissible. Inside this route, local evidence control (LEC) adjusts a support-selected interpolation scalar using endpoint margins, winner disagreement, residual direction, and local support contrast. LEC contains the matched scalar controller as a special case and is conditionally invariant to noncompeting vocabulary additions. Across seven datasets, three vision-language encoders, four support profiles, and 420 TaskRes episodes, SAFEROUTE-LEC improves class-macro accuracy over the budget-matched scalar by 1.278 percentage points under mixed support and by 1.211 to 1.402 points across all profiles. It also improves the same-loss constant by 1.048 points under mixed support, ranks first in every encoder-profile mean and every mixed-support dataset mean, retains all 173,043 protected decisions, and produces no change in 3,780 specified dummy-class tests. These results establish a modular retention interface and show that query-local evidence provides substantial utility beyond strong support-only calibration without modifying parent training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
mixed-shot recognition, vision-language adaptation, retention-constrained inference, local evidence control, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cJmpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses adaptation-induced prediction errors in mixed-shot recognition, where classes have heterogeneous numbers of labeled support examples. The authors propose SafeRoute, which preserves frozen-model predictions for unsupported classes through deterministic routing, and Local Evidence Control (LEC), which dynamically adjusts the interpolation between frozen and adapted scores using query-local evidence. The paper provides theoretical guarantees and evaluates the method across multiple datasets, vision-language encoders, and support profiles.

Critical strengths

S1. The paper addresses a meaningful problem by formalizing a prediction-retention constraint in mixed-shot recognition and providing explicit theoretical guarantees.

S2. The experimental design is well controlled. Comparisons with matched scalar and same-loss constant baselines help isolate the contribution of query-dependent control, with consistent improvements across datasets and encoders.

Critical weaknesses

W1. The contribution of the six handcrafted features in LEC is not sufficiently analyzed. Although the paper compares against fixed and count-based controls, it remains unclear whether the full feature design is necessary.

W2. Cross-parent validation remains limited. The main experiments primarily rely on TaskRes, while the additional parent-family evaluation is restricted to a single natural-history setting without detailed per-parent results.

Questions for authors

Q1. Could the authors provide feature ablations for LEC and compare it with a simpler confidence- or margin-based dynamic gating baseline under the same retention constraint?

Q2. Could the authors provide more detailed results for each parent adapter? Since the main experiments primarily rely on TaskRes, a more comprehensive cross-parent evaluation would help clarify the generalizability of SafeRoute-LEC across different adaptation methods.

Reviewer_xd64OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies vision-language model adaptation under heterogeneous support, focusing on the risk that adaptation may overturn the frozen model’s previously correct predictions for classes with no support exam- ples. The authors propose SafeRoute, which separates permission to modify a prediction from the strength of adaptation: it preserves the prediction when the frozen model’s predicted class has no support examples, and uses a local evidence controller (LEC) to adjust the interpolation between frozen and adapted scores for the remaining queries. The paper provides a necessary-and-sufficient characterization of the retention condition and an analysis of conditional locality. Across seven datasets, three encoders, and four support configurations, LEC achieves consistent average performance improvements over the paper’s matched scalar and same-loss constant controllers.

Critical strengths
  1. The experimental design is reasonably comprehensive, the experiments are sufficiently extensive, and the evaluation settings are generally appropriate. Code is also provided.
  2. The paper identifies an interesting issue in mixed-shot recognition and proposes a method to address it. Extensive experiments also demonstrate some performance improvements.
Critical weaknesses
  1. The motivation and effectiveness of the closed-route design warrant further discussion. The proposed adaptation approach includes two routes: an open route and a closed route. The closed route is intended to prevent adaptation from turning the frozen model’s originally correct predictions for zero-support classes into incorrect ones. It therefore disables adaptation whenever the frozen model predicts that a sample belongs to a zero-support class. However, this straightforward closed-route design also prevents adaptation from correcting certain errors made by the frozen model: the true label belongs to a supported class, but the frozen model predicts a zero-support class, and adaptation would otherwise correct the error. For downstream tasks requiring adaptation, the cost of forgoing these correction opportunities is not commensurate with the benefit of preserving correct predictions by disabling adaptation. Given that adaptation is intended to serve a downstream objective, why is strictly preserving all eligible frozen predictions worth the cost of blocking some corrections under that objective? Moreover, the benefits of the closed-route design depend too heavily on the predictive accuracy of the frozen model itself. Although the authors acknowledge this issue in the limitations and appendix, the motivation and effectiveness of this simple closed-route design still warrant further discussion.

  2. The substantive contribution of LEC beyond existing dynamic fusion methods has not been sufficiently clarified. There is already directly related research on dynamically adjusting the fusion strength between pretrained and adapted models based on query information, such as DaWin and VRF. The paper has clear distinguishing features in its strict pointwise retention constraint and routing design. However, it does not sufficiently discuss how the specific LEC controller relates to, and differs from, existing dynamic fusion methods. In particular, the paper needs to explain more clearly how local competition features, support counts, and fitting the controller using only the support set address the specific constraints of mixed-shot recognition, and what additional contribution these choices provide. Comparisons against the matched scalar and same-loss constant support the effectiveness of the proposed controller under the paper’s evaluation protocol, but do not fully establish its distinct contribution relative to existing research on adaptive fusion. I suggest that the authors add a comparison of the underlying mechanisms in terms of available information, fusion formulations, fitting requirements, and retention guarantees, and clearly distinguish the respective contributions of the strict retention framework and the LEC controller.

Questions for authors
  1. Could the authors report, for the experiments involving zero-support classes, the number of correct predictions for zero-support classes that the closed route saves from being overturned by adaptation, as well as the number of incorrect frozen-model predictions that adaptation would otherwise correct but that remain uncorrected because of the closed route?

  2. Could the authors briefly justify disabling adaptation whenever the frozen model predicts a zero-support class? Given that adaptation is intended to serve a downstream objective and correct the frozen model’s prediction errors, does this design conflict with the original purpose of adaptation? Preserving correct predictions for zero-support classes is important. However, for the target downstream task, is guaranteeing retention of the frozen model’s originally correct predictions for zero-support classes worth forgoing opportunities to correct errors on some samples from supported classes? Specifically, I refer to cases in which the frozen model misclassifies a sample from a supported class as belonging to a zero-support class, and adaptation would otherwise correct the error.

Reviewer_J7DvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies adaptation of vision-language models (VLMs) in mixed-shot settings where the number of support samples vary across classes while still having access to text labels for all classes. The paper focus on determining whether frozen model or the adapted one is a better choice for inference. The paper proposes a lightweight score interface method using the frozen model for unsupported classes and performs residual interpolation for the others.

Critical strengths

The paper explores a realistic mixed-shot setting using vision-language models which raises an interesting problem of whether to use the adapted or frozen model scores for individual classes.

Critical weaknesses
  1. Poorly written paper. Several terms (Global count schedules, parent residual, endpoint-local margins and more) are not defined. Many statements are unclear and disconnected. The abstract is poorly written with no background and definitions of terms like heterogeneous support, mixed-shot setting.

  2. Poor contributions: the proposed method offers a rather trivial solution of interpolating scores from adapted and frozen model for supported classes. The method is not validated against any existing approaches.

Questions for authors

See weaknesses for my questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.