acceptodds
Desk rejected at ICLR 2027

TailSafe: Decision-Relevant Certification for Long-Tailed Adaptation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Prior correction can improve a long-tailed classifier's overall accuracy while degrading a frequency group. TailSafe determines whether the available evidence justifies executing a correction under baseline-relative overall and group risk budgets. Its key observation is that budget feasibility can be resolved without uniquely recovering the target prior. Frozen logit and prototype witnesses identify compatible environments through mixture-direction concentration. A certification-aware selector anticipates an independent paired-loss audit whose uncertainty adapts to error-change second moments. An intersection–union rule controls erroneous release without a direction-count penalty in the audit threshold. Under label shift, a declared finite prior family, and independent calibration, TailSafe provides finite-sample risk control and protected preservation. An exact ambiguity dual and a nonasymptotic execution bound separate action-relevant prior uncertainty from directional source-evidence requirements. Experiments span ten frozen learners and four image benchmarks. In controlled CIFAR-100-LT deployment, TailSafe achieves a 2.4 percentage-point overall gain with 70% execution and 0.4% test-estimated budget exceedance. Finite-support studies demonstrate useful execution under exactly indistinguishable priors. Code is available at Supplement.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
long-tailed recognition, label shift, selective adaptation, risk certification, finite-sample guarantees

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LupaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TailSafe, a framework for safely adapting long tailed classifiers under uncertain deployment distributions. The main idea is to determine whether a proposed correction satisfies overall and group risk constraints without requiring exact recovery of the target class prior. The method combines prior compatibility estimation, independent risk auditing, and certification aware selection. The authors establish finite sample risk guarantees and evaluate the approach across four image benchmarks and ten classifiers, showing improvements in accuracy while maintaining low rates of budget violations.

Critical strengths

The paper addresses an important problem in long tailed classification. I particularly like the observation that identifying the exact target prior is not necessary when the goal is simply to certify whether an adaptation satisfies risk constraints. The theoretical analysis provides a useful formalization of this idea.

The framework provides finite sample guarantees while allowing useful adaptation rather than always preserving the baseline. The experiments are fairly comprehensive, covering multiple datasets and classifiers. The results in Tables 3 and 4 are particularly encouraging, showing meaningful accuracy improvements with substantially lower risk of budget violations.

Critical weaknesses

The theoretical guarantees rely on restrictive assumptions, particularly exact label shift and a prespecified family of target priors. Although the appendix considers prior misspecification, the execution rate decreases considerably as the allowed uncertainty increases. It is unclear how useful the framework would remain under more realistic deployment shifts where class conditional distributions may also change.

The risk guarantees are defined at the level of predefined frequency groups rather than individual classes. Consequently, a correction could satisfy the group budget while substantially degrading performance for particular rare classes. This is an important limitation given that protecting minority classes is one of the main motivations for long tailed adaptation. The paper would benefit from a clearer analysis of this tradeoff.

Questions for authors

How should the candidate prior family be constructed in a practical deployment where the target distribution is unknown? How sensitive is the execution rate to the size and coverage of this family?

Can the authors provide additional insights into how TailSafe behaves under mild violations of the label shift assumption, beyond the controlled sensitivity analysis in the appendix?

Have the authors examined the maximum deterioration of individual rare classes among the accepted corrections? This would help clarify what protection the group level guarantees provide in practice.

3 comments

Thank you for your constructive review and for recognizing the value of decision-relevant identification and useful adaptation with finite-sample control. We address each weakness and question below, including the requested maximum deterioration of individual rare classes.

W1:

We agree that coverage and usefulness must be considered together. TailSafe's guarantee addresses uncertainty in the target prior under stable class-conditional distributions; within that setting, it can certify a correction without uniquely recovering the prior. Appendix E separately handles a prespecified amount of prior mismatch, but it does not cover arbitrary changes in class-conditional input distributions.

The observed loss of execution under broader uncertainty is real, but useful adaptation remains in the reported settings. For example, expanding the dictionary from 7 to 56 templates retains 57% execution and a +1.9 pp Overall gain (Table S26). The neighborhood configurations in Table S27 retain 49% and 30% execution with +1.4 and +0.8 pp gain at allowed radii 0.02 and 0.05. These results show an explicit coverage–utility trade-off, not a claim that any uncertainty range will remain useful.

Q1 below addresses practical family construction and dictionary sensitivity. Q2 gives the additional image-change diagnostic, including a stronger-change condition where the nominal risk target is not maintained. We keep these empirical observations separate from the theorem's coverage.


W2:

We agree with this distinction. The default certificate protects overall and prespecified group risks, not each class individually. A positive average Few gain cannot answer your concern. We therefore report the requested execution-conditioned classwise diagnostic under Q3, including the maximum observed decline rather than only averages.

The audit principle is not restricted to three frequency groups. A class cc specified before auditing can be represented by the singleton direction w=ecw=e_c with the additional check

Lec(a;δr)+βc≥0.L_{e_c}(a;\delta_r)+\beta_c\geq0.

Here ece_c is the class-cc unit vector and βc\beta_c is its prespecified deterioration budget. The independent all-tests-pass argument extends to such constraints when the candidate is fixed before audit access. This is an optional policy, not part of the default guarantee.

The principal cost is evidence at the requested granularity. Intersection–union testing (IUT) does not introduce a Bonferroni penalty over these directions, but individual-class checks cannot pool observations across classes. For illustration, under the paper's radius formula, δr=0.025\delta_r=0.025, equal observation weights, no missing mass, and zero observed paired changes, 40 observations of one class give a radius of approximately 39.90 pp. A direction uniform over 35 classes with 40 observations each gives approximately 1.14 pp (Appendix D.4). This is a calculation for the current auditor, not a universal sample-complexity lower bound.

Appendix F's Macro-Few policy remains an aggregate and is not a substitute for classwise protection. The measured singleton-policy result is included under Q3 so that both residual harm and the cost of finer checks are visible.

Q1:

The family should be fixed before deployment using source statistics and independent knowledge of plausible operating conditions. Appendix G.3 includes source-frequency, uniform, reverse-frequency, interpolated, and group-heavy priors. The target batch identifies compatible members rather than treating one prior as known.

Table S26 adds distinct templates while holding the generating-prior collection fixed.

Dictionary sizeExecution (%)ΔAll\Delta\mathrm{All} (pp)Test-estimated Exc. (%)
7702.40.4
14652.20.4
28622.10.5
56571.90.4

Execution decreases from 70% to 57% as plausible explanations increase, while gain remains positive. Dictionary size enters identification calibration, and more retained constraints can reduce execution power even without a direction-count factor in the marginal audit threshold.

This size sensitivity does not establish coverage of arbitrary unknown priors. For bounded off-template targets, Appendix E provides a separate procedure with a prespecified neighborhood radius. A nonempty compatible set verifies neither prior coverage nor label shift. Historical labeled deployment data or independent target-side validation can help assess these assumptions; compatibility cannot replace either.


Q2:

Yes. Table S27's conditional-mixture construction leaves the unlabeled input marginal unchanged and is evaluated exactly. It demonstrates an identification limit, not neural-image robustness. The separate image diagnostic below uses 1,000 decisions per condition. Audited gains compare the returned predictor with the unchanged baseline under the same condition.

Table R3. Additional target-image diagnostic.

Target conditionDirect-execution Exc. (%)Audited execution (%)Audited ΔAll\Delta\mathrm{All} (pp)Audited exceedances / 1,000
Original images6.068+2.25
Mild brightness change8.063+1.812
Mild blur11.257+1.226
Stronger conditional change19.645+0.363

The tested mild changes retain useful adaptation with lower observed exceedance than direct execution. The stronger change reaches 6.3% all-decision exceedance, or 14.0% among the 450 executed corrections, so empirical benefit remains dependent on model assumptions.

These diagnostics are separate from the main controlled-deployment results. They do not extend Theorem 1 to conditional shift or supply the same guarantee conditional on execution. Returning the baseline avoids modification-induced harm, but does not ensure its accuracy after an environment change.


Q3:

Yes. Let Δj,c\Delta_{j,c} be the test-estimated class-cc accuracy change under executed correction jj, in percentage points. We compute

Wj=max⁡c∈GFewmax⁡0,−Δj,cW_j=\max_{c\in G_{\mathrm{Few}}}\max\\{0,-\Delta_{j,c}\\}

for each executed correction before summarizing its distribution. Averaging across decisions first could conceal a severe individual execution. The final column uses Few-class/executed-decision pairs as its denominator.

Table R4. Additional classwise diagnostic. Gains and deterioration are in percentage points.

Audit policyExecution (%)All-decision ΔAll\Delta\mathrm{All}Median WjW_j95th percentile of WjW_jMaximum class declineFew pairs declining by more than 2 pp (%)
No final risk audit80+2.89192821
Group-level risk audit69+2.36132012
Group audit + checks for every Few class00.0————

These results are separate from Table 4. Execution and Overall gain use all decisions; the deterioration columns condition on execution. Dashes indicate no executed corrections, not zero measured classwise harm.

The maximum observed individual-class decline under the group-audited policy is 20 pp. Its median worst-class decline is 6 rather than 9 pp, its 95th percentile is 13 rather than 19 pp, and 12% rather than 21% of Few-class/decision pairs decline by more than 2 pp. It retains 69% execution and a +2.3 pp all-decision gain.

These subset comparisons are descriptive, not same-subset causal estimates or population worst-class guarantees. The results show both useful screening and residual classwise harm. Checking every Few class prevents execution in the tested configuration; this does not imply that per-class certification is impossible with other evidence or auditors.

Thank you for prompting this finer-grained analysis. The responses quantify the protection target, residual classwise variation, and evidence costs of broader coverage and finer protection. We would appreciate your reassessment of the decision-relevant certification contribution in light of these results.

AuthorsOpenReview ↗

Thank you for your thoughtful feedback

Thank you again for your positive assessment of our work and for your thoughtful questions. We hope that our responses and additional analyses help address your concerns about the deployment assumptions and the distinction between group-level and classwise protection. We sincerely appreciate your careful review and would be happy to provide any further clarification.

Reviewer_ezSMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TailSafe, a framework for deciding whether to apply a correction to a frozen long-tailed classifier. Instead of estimating a single target prior, the method identifies a set of compatible priors and checks whether a correction satisfies overall and group-level risk budgets. The approach combines prior identification, correction selection, and an independent paired-loss audit. The paper provides finite-sample guarantees under label shift and evaluates the method on several long-tailed image classification benchmarks.

Critical strengths
  1. The distinction between recovering the target prior and deciding whether a correction is acceptable makes sense. Different priors may lead to the same decision, so exact prior estimation is not always necessary. The analysis in Section 4 provides a useful explanation of this idea.
  2. Separating candidate selection from the final audit avoids using the same data for both decisions. The paired-loss formulation also makes sense because the goal is to measure the change caused by a correction. The results in Table 3 support the usefulness of the audit. For the same LSC proposals, the reported budget exceedance rate drops from 6.0% to 0.5% after applying TailSafe, although fewer corrections are executed.
Critical weaknesses
  1. The main results assume exact label shift and that the true target prior belongs to a predefined finite set. I am not sure how often these conditions can be justified in practice. The authors discuss this limitation in Appendix K and include sensitivity experiments in Table S27. These experiments are useful, but they also show what can happen outside the assumed setting. For example, the conditional-mixture experiment reports a 10.8% budget exceedance rate at a mixing level of 0.15. This is outside the theoretical guarantee, but it makes me somewhat cautious about the practical scope of the method.
  2. TailSafe needs separate source data for identification, selection, and auditing. This is reasonable for the theoretical guarantee, but the cost may be significant in long-tailed settings. For CIFAR-100-LT, 100 images per class are reserved before constructing the training set. Table 6 also shows that, with only 25 calibration images per class, the execution rate drops to 18% and the overall gain is just 0.5 percentage points. The matched-checkpoint comparisons are useful, but I would like to know whether reserving these labels for certification is better than using them to train the original classifier. This trade-off is not entirely clear from the current results.
Questions for authors
  1. How would TailSafe be used when the target prior is not covered by the predefined set, or when the label-shift assumption is only approximately satisfied? The experiments in Table S27 help, but I would like to understand how these assumptions could be assessed in a real deployment.

  2. Have the authors compared TailSafe against a classifier trained with the full source dataset, without reserving data for certification? Such a comparison would help clarify whether the benefits justify the additional labeled-data requirement.

3 comments

Thank you for your thoughtful review and for recognizing the distinction between prior recovery and decision-relevant certification, as well as the value of independent auditing. We respond to the two weaknesses and two questions in their original order; overlapping concerns are linked explicitly rather than left unanswered.

W1:

We agree that these assumptions require justification. They specify which uncertainty the certificate controls: under stable class-conditional distributions, TailSafe can certify a correction despite unresolved uncertainty about the target prior. It does not require that prior to be uniquely identified.

Two different departures from the main setting should be distinguished. Bounded prior mismatch has a separately certified extension in Appendix E. In Table S27, allowed neighborhood radii of 0.02 and 0.05 yield 49% and 30% execution, +1.4 and +0.8 pp Overall gains, and 0.6% and 0.5% test-estimated exceedance. Broader declared coverage reduces execution but still permits useful adaptation in this study. The radii are permitted mismatch ranges, not two measured target-shift magnitudes.

Class-conditional change is not covered by that extension. The 10.8% result is an exact evaluation of the specified conditional-mixture model, not a neural-image corruption measurement. In that construction, the unlabeled input marginal is unchanged, so those observations cannot reveal the change in the labeled target law. This result identifies a genuine limitation, not a guarantee that remains valid after label shift fails. Under Q1, we explain how this distinction affects deployment and provide the additional image-change diagnostic.


W2:

Thank you for emphasizing this trade-off. The independent audit provides information that was not used to select the correction; this is essential to the stated certification argument, but it has a real data cost. Table 6 shows the evidence–utility relationship at a fixed model: using 25 rather than 100 calibration images per class reduces execution from 70% to 18% and the Overall gain from 2.4 to 0.5 pp. Unused reserved images in that study do not return to training, so it does not answer the opportunity-cost question by itself.

Table S8 also has a narrower purpose: it compares label allocation within post-hoc fitting, not full classifier retraining. We address your requested training comparison directly under Q2. It shows that using all original training and calibration images for retraining yields 60.0% accuracy, compared with 52.0% for the reserved-protocol model with TailSafe. Thus, we do not claim that reserving labels is accuracy-optimal when unrestricted retraining is available. The contribution is the independent certification capability for modifications of an already fitted model; the additional comparison quantifies its cost.


Q1:

The prior family should be fixed before deployment using source statistics and independently available information about anticipated operating conditions. Appendix G.3 uses source-frequency, uniform, reverse-frequency, interpolated, and group-heavy templates. These are declared environments, not universal coverage of unknown deployments.

Within a prespecified prior neighborhood, the Appendix E procedure enlarges identification tolerance and tightens release checks. For a retained positive-mass group, the check is

q(g)[Lwg(q)(a;δr)+βg]≥επ.q(g)\bigl[L_{w_g(q)}(a;\delta_r)+\beta_g\bigr]\geq\varepsilon_\pi.

Here q(g)q(g) is the template's group mass, Lwg(q)L_{w_g(q)} is the audited gain lower bound, βg\beta_g is the group budget, and επ\varepsilon_\pi is the declared neighborhood radius. Broader coverage consumes audit margin, especially for low-mass groups. The procedure still assumes exact label shift.

A nonempty compatible set does not establish prior coverage or validate label shift. Historical labeled deployment information and independent target-side validation can help assess applicability, but compatibility alone cannot exclude every class-conditional change. If the declared conditions cannot be justified, the source-based certificate should not be represented as a target-risk guarantee. Additional target-side validation is an external safeguard, not an automatic capability of the method.

Complementing the constructed model in Table S27, the image diagnostic reports 1,000 decisions per condition. Audited gains compare the returned predictor with the unchanged baseline under the same target condition.

Table R3. Additional target-image conditional-change diagnostic.

Target conditionDirect-execution Exc. (%)Audited executions / 1,000Audited ΔAll\Delta\mathrm{All} (pp)Audited exceedances / 1,000Exc. among audited executions (%)
Original images6.0680+2.25 (0.5%)0.74
Mild brightness change8.0630+1.812 (1.2%)1.90
Mild blur11.2570+1.226 (2.6%)4.56
Stronger conditional change19.6450+0.363 (6.3%)14.00

These records are separate from the main controlled-deployment results and the exact construction in Table S27. The tested mild changes retain positive utility with lower observed exceedance than direct execution. The stronger change reaches 6.3% all-decision exceedance, showing an empirical limit. These observations do not extend Theorem 1 beyond label shift; the execution-conditioned percentages are descriptive and do not inherit its unconditional failure bound.


Q2:

Yes. The additional CIFAR-100-LT IR=100 comparison distinguishes standard unreserved LT training from training on our pipeline's entire image budget. Let TT be the reserved-protocol training set and HH the 100-per-class calibration set. The reference Few subset is fixed across configurations, so training-count changes do not alter the evaluated classes.

Table R2. Training–certification allocation comparison.

ConfigurationTraining imagesImages used for calibrationAll (%)Reference Few (%)
BCL trained on TT; no adaptation8,668050.733.9
Same BCL + TailSafe8,66810,00052.039.5
Standard unreserved LT training, Nmax⁡=500N_{\max}=50010,847052.335.9
Retraining on T∪HT\cup H18,668060.051.0

The first two rows reproduce Table 1; the last two add training comparisons. Standard unreserved LT construction is not a total-label-budget match to T∪HT\cup H. The latter adds 100 images to every original class, raises the smallest training count from 4 to 104, and changes the imbalance profile. It uses the full image budget of the reserved-protocol pipeline, not the entire balanced CIFAR-100 parent pool.

Relative to standard unreserved LT training, TailSafe is 0.3 pp lower in Overall accuracy and 3.6 pp higher on the fixed reference Few subset. Full-budget retraining is substantially more accurate: 60.0% versus 52.0%. When retraining is feasible and accuracy is the sole objective, using all these labels for training is preferable in this comparison.

This quantifies the resource trade-off while retaining the distinct value of certification. The matched-checkpoint and identical-proposal experiments ask whether a modification of an existing classifier can be independently justified; Table R2 asks whether the same images could instead fit a stronger classifier. We distinguish these capabilities without claiming either always dominates.

Thank you for helping clarify these practical choices. The evidence supports a decision-relevant certificate for frozen-model adaptation, with measured costs and explicit coverage conditions. We would be grateful for your reassessment in light of the direct resource comparison and additional deployment diagnostics.

AuthorsOpenReview ↗

Thank you for your thoughtful feedback

Thank you again for your positive assessment of our work and for your constructive feedback. We hope that our rebuttal and additional results help address your questions about the scope of the guarantees and the practical trade-offs in allocating labeled data. We sincerely appreciate your time and consideration and would be happy to clarify any remaining points.

Reviewer_mxs9OpenReview ↗
Rating
3(+2)
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper provides a method to correct the results of a classifier in the case of a low number of samples in a class and where retraining is not possible. The method, coined TailSafe, has been explained and has been validated in a set of experiments involving state-of-the-art data sets en models. The results show a improvement of the classification performes in low-frequent classes, but this comes at a cost of the accuracy in the high-frequent classes.

Critical strengths

S1: the method is very relevant and is supported by strong mathematical reasoning.

Critical weaknesses

W1: I had a lot of difficulty to follow the mathematics and the reasoning. Allthough I think the article is interesting, it would require an enormous effort to understand the paper. The paper has also a supplement of about 45 pages, which suggests that this work is too large to be concise.

W2: A lot of terminology and abbreviations are not explained. E.g. the text mentions IR, but it is nowhere to be found what IR means.

Questions for authors

Q1: In Table one, the accuracy in the few classes increases a lot, while the accuracy in the many and medium classes decreases. Yet the overal accuracy increases. How did the Few/Medium en many classes devided. If you have a lot of few classes than you have an improvement of each of those. This accumulates the improvement, which overestimates the improvement?

3 comments

Thank you for your time and for recognizing the relevance of the problem and its mathematical motivation. We appreciate your feedback on accessibility and evaluation, and respond to each weakness and question below.

W1:

Thank you for highlighting this. We agree that readers should be able to understand the problem and the main idea without navigating the full appendix. A self-contained explanation is as follows.

TailSafe asks whether the evidence justifies releasing a particular correction of a frozen classifier. It first retains target priors compatible with the observations. It then fixes one correction and uses independent labeled source data to assess the correction's error changes relative to the original classifier. The correction is executed only when all required overall and group budget checks pass; otherwise, the original classifier is returned unchanged.

The key observation is that several priors can remain plausible while still agreeing that the same correction satisfies the budgets. Thus, uniquely recovering the target prior can be unnecessary for deciding whether to execute the correction. Sections 3.1 and 3.4 define this decision, Theorem 1 establishes risk control, and Section 4.2 characterizes decision-relevant prior ambiguity. The appendix provides complete proofs, protocols, and additional diagnostics to support verification, rather than replacing this central explanation.

The same-proposal experiment makes the distinction concrete. Applying TailSafe to identical LSC proposals reduces test-estimated budget exceedance from 6.0% to 0.5%, while retaining a +1.9 pp Overall gain and 60% execution (Table 3). The contribution is therefore an independently calibrated execution decision, not simply a different correction that favors Few classes.


W2:

We agree that IR should have been expanded at its first occurrence. IR means imbalance ratio, the parameter controlling the largest-to-smallest class training-count ratio in the benchmark construction (Appendix A.1). IR=100 therefore denotes a substantially more imbalanced training setting than IR=10.

For clarity, pp means percentage points; Int. is the execution rate over all declared deployment decisions; Exc. is the test-estimated frequency with which at least one overall or group budget is exceeded, also over all declared decisions. IUT means intersection–union testing: a correction is released only when every required check passes. We explain the frequency-group definitions and their evaluation weights directly under Q1 below.


Q1:

Thank you for asking us to make the calculation explicit. Groups use the actual training counts after calibration reservation: Many contains classes with Nc>100N_c > 100, Medium those with 20≤Nc≤10020 \leq N_c \leq 100, and Few those with Nc<20N_c < 20. Membership is fixed before adaptation and evaluation. “Few” refers to limited training examples per class, not to the number of classes in that group.

Table 1 reports class-uniform Top-1 accuracy: every class receives equal weight. The three groups are not equally weighted, and no additional weight is assigned to Few-class improvements. At IR=100, the groups contain 30, 35, and 35 classes (Table S3). The BCL example decomposes as follows.

Table R1. Aggregation of the BCL results in Table 1.

GroupClassesBaseline (%)TailSafe (%)Change (pp)Contribution to Overall (pp)
Many3067.666.1-1.5-0.450
Medium3553.052.3-0.7-0.245
Few3533.939.5+5.6+1.960
Total100———+1.265, approximately +1.3

ΔAll=0.30(−1.5)+0.35(−0.7)+0.35(5.6)=1.265pp≈1.3pp.\Delta\mathrm{All}=0.30(-1.5)+0.35(-0.7)+0.35(5.6)=1.265\\ \mathrm{pp}\approx1.3\\ \mathrm{pp}.

This illustration uses displayed, rounded group values; the reported metrics are aggregated before rounding. A larger group contributes more because it contains more equally weighted classes, not because its improvement is counted again. Both predictors use exactly the same weights and evaluation measure. Appendix H provides the aggregation identity. For nonuniform deployment priors, the corresponding weights are the target group masses, not necessarily these class-uniform weights.

The group decreases also need to be interpreted relative to the stated protection target. The default settings use a zero overall deterioration budget and a 2 pp group budget. Theorem 1 controls budget exceedance for the complete decision under its assumptions; it does not require every displayed group accuracy to improve. Execution is determined by the independent audit, not retrospectively by the rounded test accuracies.

We hope these explanations resolve the aggregation concern and make the certification contribution more accessible. We would be grateful for your reassessment in light of the explicit calculation and the matched-proposal evidence.

Reviewer_mxs9OpenReview ↗

Appreciation of the comments

Dear authors,

Thank you for your clear explanation. I also reviewed the remarks of the other reviewers and the rebuttal. I therefore raise my score to 3.

greetings,

AuthorsOpenReview ↗

Thank you for your thoughtful consideration

Thank you for taking the time to revisit our submission and consider our clarifications alongside the other reviewers’ comments. We are glad that our explanations were helpful, and we sincerely appreciate your thoughtful engagement in the discussion and your updated assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Reviewer Responses — Submission 16649

Dear Area Chair,

Thank you for overseeing the review of TailSafe: Decision-Relevant Certification for Long-Tailed Adaptation. We briefly summarize the contribution and our responses to the three reviewers.

Contribution. TailSafe studies whether a correction of a frozen classifier can be executed under overall and predefined group risk budgets when the target prior is uncertain. It separates constructing a promising correction from establishing that its deployment is justified: higher overall accuracy alone does not ensure that group budgets are satisfied. The central observation is that certifying a particular correction need not require uniquely recovering the target prior. Compatible-prior identification, certification-aware selection, and an independent paired-loss audit implement this distinction, with preservation of the original classifier when the required evidence is insufficient.

The theoretical analysis establishes finite-sample risk control under the stated label-shift, prior-coverage, and independent-calibration assumptions, and gives sufficient conditions for useful execution that distinguish prior ambiguity from uncertainty due to limited source evidence. The experiments evaluate recognition across ten frozen learners and four image benchmarks, with separate controlled deployment studies measuring execution and test-estimated budget exceedance. Identical-proposal comparisons isolate the certification layer's effect, while finite-support studies demonstrate useful execution under priors with identical input marginals. These results examine both risk control and the ability to make beneficial changes rather than always preserving the baseline.

Reviewer mxs9. We provided a plain-language explanation of the method, defined IR and the evaluation terminology, and worked through an explicit calculation of Overall improvement. This clarifies that the baseline and corrected predictor use the same fixed class weights, without additional weighting of Few-class gains. We also distinguished improving Few accuracy from independently certifying whether a correction should be executed.

Reviewer ezSM. We explained the difference between exact-family coverage, bounded prior mismatch, and class-conditional change, including what unlabeled compatibility can and cannot establish. We reported a direct training–calibration allocation comparison, acknowledging the accuracy advantage of allocating all available labels to retraining in that experiment. Additional image-change diagnostics characterize the empirical benefits and limits of auditing outside exact label shift.

Reviewer Lupa. We reported prior-family-size sensitivity and the requested classwise deterioration statistics among executed corrections, including the maximum observed rare-class decline. We clarified that the default group-level guarantee does not imply protection of every class, and examined the execution cost of adding classwise checks. The image-change diagnostics also address the request for evidence beyond the constructed conditional-mixture example.

These responses provide a more explicit account of the method's contribution and practical trade-offs, without extending its guarantees beyond their assumptions. We hope they assist your assessment of the paper's technical results and experimental evidence. Thank you for your time and consideration.

Best regards,
The Authors of Submission 16649

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.