acceptodds
Under review as a conference paper at ICLR 2027

REAP: Rare-event Adaptation for Deep Learning

22 views

Abstract

Rare but important samples are critical to model reliability, yet their low frequency creates a persistent rare-sample underfitting problem: standard training is dominated by common patterns, causing rare samples to be systematically underrepresented and poorly learned. Existing importance sampling strategies can increase the exposure of informative samples during post-training, but they do not modify the model architecture to help pretrained models adapt to rare samples, motivating an architectural solution for more effective rare-sample adaptation. To address this challenge, we propose REAP, a rarity-aware adapter that can be embedded into pretrained backbone networks during post-training to facilitate adaptation to rare samples. REAP is applicable to multi-stage hierarchical backbones and introduces lightweight residual adaptation, enabling a smooth transition from pretrained representations to post-training. Furthermore, we propose an implicit routing mechanism, where continuous spatial gates modulate adapter residuals through element-wise multiplication. This design enables smooth and stable adaptation without requiring additional supervision signals or rule-based preprocessing, thereby supporting fully end-to-end post-training. REAP can also be naturally combined with existing importance sampling methods, yielding complementary benefits from architectural adaptation and informative-sample sampling. Extensive experiments with a SegFormer backbone across semantic segmentation, medical image classification, and regression tasks demonstrate the effectiveness of REAP. Compared with the corresponding no-Adapter baselines, REAP improves tail IoU on ADE20K by 10.64%, increases classification accuracy on ISIC by 3.64%, and reduces the PHM08 score on the FD001 dataset by 7.12%, while preserving overall performance, contributing to safer and more trustworthy AI systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Rare-Sample Learning, Parameter-Efficient Adaptation, Implicit Routing, Importance Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rSnbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Rare but important samples often receive weak gradient signals during training, leading models to underfit them. This paper proposes REAP, an adapter module inserted after each stage of a hierarchical backbone, to provide architectural support for learning from rare samples. The method is evaluated across multiple task domains and shows improved tail IoU on ADE20K.

Critical strengths

The evaluation covers a broad range of tasks, including segmentation, classification, and regression.

Critical weaknesses
  1. Single-run results and a weak baseline. The paper appears to report only one run per experiment, making it unclear how much run-to-run variance could affect the conclusions. This is especially concerning when the improvements over the baselines are tiny (e.g., Kvasir F1: SegFormer 93.75 vs. REAP+IS 93.76). Moreover, the baseline SegFormer (MiT-B4) achieves only 24.1 mIoU on ADE20K, far below the 50.3 mIoU reported by Xie et al. (2021). Although the experimental protocols differ, gains over such a weak baseline are difficult to interpret, making the improvements less convincing.

  2. Limited experimental breadth and insufficient validation of REAP's architectural contribution. The proposed module is evaluated only on variants of SegFormer, despite being intended as a broadly applicable method. Moreover, the current experiments do not sufficiently isolate the effects of different components. I would expect the following ablations on one or two key datasets:

No adapterPlain adapter (with approximately the same number of parameters as REAP)REAP
Without IS✓✓✓
With IS✓✓✓

The method combines known components (importance sampling, bottleneck layers, and sigmoid gating), with REAP being the main contribution. However, it remains unclear why REAP's architecture would specifically help the network learn more effectively from rare samples.

Questions for authors
  1. ImageNet-LT is a standard benchmark for long-tailed learning. Why do you construct a three-class ImageNet subset instead of using ImageNet-LT?

  2. How can you demonstrate that the improved downstream performance comes from REAP's design rather than simply from additional trainable parameters? For example, on ADE20K, the backbone learning rate is 5e-7, while the adapter learning rate is 5e-5. This effectively introduces additional trainable parameters that are updated with a much higher learning rate than the backbone. Therefore, a parameter-matched plain adapter baseline would help clarify whether the improvements come from REAP's specific architecture or simply from the additional trainable capacity.

3 comments
AuthorsOpenReview ↗

Response to Baseline Evaluation, Architectural Ablation, and Long-Tailed Benchmark Design

We thank the reviewer for the detailed feedback. We have revised the manuscript with additional ablations and analyses to better clarify the contribution and scope of REAP.

1. ADE20K baseline and evaluation protocol.

Our ADE20K experiments use a crop-based protocol with 256 × 256 crops for both training and validation (Appendix A.2.1), which differs from the original SegFormer evaluation setting. Thus, the absolute mIoU values are not directly comparable. Importantly, all methods in Table 3 follow the same protocol. Within this controlled setting, REAP improves overall mIoU from 24.14% to 30.61% and tail IoU from 9.94% to 20.58%, while preserving head-category performance. We acknowledge that evaluation under the original high-resolution protocol would provide stronger evidence of general applicability.

2. Architectural contribution and component ablation.

We introduce the requested ablation structure in Table 8, comparing No Adapter, Plain Adapter, and REAP under both uniform and importance sampling on C-MAPSS FD001/FD004. The Plain Adapter retains the same residual bottleneck architecture and training protocol but removes input-dependent spatial gating. Under IS, REAP improves PHM08 on both subsets and achieves the lowest RMSE on FD004. This comparison provides evidence of a gating-specific benefit beyond introducing a conventional adapter with additional trainable capacity.

We also provide an instance-level rarity analysis (Figure 3), where gate activation correlates positively with visual rarity (Spearman's ρ = 0.717), including after within-class centering (ρ = 0.494). Figure 1(c) further illustrates distinct adaptation behaviors across feature stages. Together, these analyses provide additional evidence supporting the intended input-dependent adaptation mechanism.

3. Choice of the ImageNet subset instead of ImageNet-LT.

We constructed the three-class ImageNet subset to examine post-training under a fixed, severe class imbalance that persists at test time. It contains one frequent class (tench) and two visually distinct minority classes (conch and Labrador retriever), each representing approximately 1% of both training and test examples (Appendix A.1.2, Table 10).

Preserving this imbalance is important because precision at a fixed high recall depends on class prevalence. Our setting therefore examines whether REAP can recover minority-class examples while limiting false positives under the same imbalance encountered during training (Table 4).

ImageNet-LT addresses a different, broader setting: recognition across 1,000 classes with varying training frequencies and a class-balanced test set. We acknowledge that our three-class experiment does not establish performance at that scale and is intended as a targeted evaluation of high-recall minority-class recognition rather than a replacement for ImageNet-LT.

4. Scope of architectural generalization.

Although our experiments cover classification, segmentation, and regression, the current implementations use variants of the SegFormer family. We therefore distinguish generalization across task domains from generalization across backbone architectures. Extending REAP to other hierarchical backbones remains an important direction for further validation.

We hope these clarifications and the additional analyses address the concerns regarding the experimental setting and the contribution of REAP's architectural design.

Thank you to the authors for the response and the additional experiments. Having reviewed them, I maintain my original rating.

The newly added Table 8 provides the exact comparison I requested; however, it reveals that the gating mechanism offers almost no additional benefit over the Plain Adapter. Under the IS setting, REAP’s PHM08 score is only about 0.2% lower than that of the Plain Adapter (397.50 vs. 398.28 for FD001) and 0.4% lower (13198.49 vs. 13250.65 for FD004). For FD001, the RMSE is nearly identical to that of the Plain Adapter (15.0316 vs. 15.0295)—and it should be noted that these figures represent the results of only a single run. In contrast, for FD004, the PHM08 score improved from 14716.48 (without an adapter) to 13250.65 (with the Plain Adapter). At least on this dataset, the performance gains appear to stem primarily from the introduction of additional trainable parameters, rather than from the gating design itself.

Furthermore, my question regarding the variance across multiple runs remains unaddressed.

AuthorsOpenReview ↗

Clarification of Spatial Gating and Multi-Seed Evaluation

We thank the reviewer for the careful follow-up and for highlighting the limited incremental performance gains of gating and the lack of multi-run variance estimates. We agree that the original single-run results in Table 8 are insufficient to establish a robust performance advantage of the gating mechanism. We have therefore repeated the experiments using three random seeds (42, 43, and 44).

Multi-seed evaluation. All experiments follow the original IS/Top-K protocol, using the same pretrained checkpoint while varying the randomness of post-training. No subsequent changes to the learning rate, gating architecture, or training procedure were introduced. For each run, we select the checkpoint with the best validation PHM08 score and report the corresponding test results. The mean and sample standard deviation across three seeds are summarized below.

DatasetMethodRMSE ↓PHM08 ↓
FD001No Adapter15.1305 ± 0.0034405.10 ± 0.46
FD001Plain Adapter15.0351 ± 0.0130398.83 ± 0.55
FD001REAP15.0389 ± 0.0065398.75 ± 1.27
FD004No Adapter24.8334 ± 0.000014716.48 ± 0.00
FD004Plain Adapter24.7017 ± 0.004913270.84 ± 39.59
FD004REAP24.6971 ± 0.011113241.78 ± 55.90

For FD004, all three No-Adapter runs selected the shared initialization checkpoint at step 0, which explains the zero standard deviation.

Across three seeds, REAP consistently improves upon the No-Adapter baseline in both datasets, reducing the mean PHM08 score by 1.57% on FD001 and 10.02% on FD004. However, we acknowledge the reviewer's central observation: the additional PHM08 improvements over the Plain Adapter are modest in the regression tasks. These results indicate that the additional residual adaptation capacity accounts for a substantial portion of the aggregate performance gains in these regression experiments. We do not interpret the multi-seed results as evidence of a statistically robust standalone performance advantage from gating.

Due to time constraints, we have not yet evaluated the visual tasks across multiple seeds. As a complementary qualitative analysis, we visualized gate activations from the trained ISIC checkpoint and observed spatially heterogeneous, stage-dependent responses. These visualizations characterize the learned modulation patterns of the gating mechanism.

Clarification of the gating mechanism. We would also like to clarify the motivation behind the spatial gate. Its primary purpose is not simply to maximize aggregate predictive performance, but to introduce an input-dependent, regularization-like inductive bias into residual adaptation. Specifically, the learned gate continuously modulates the magnitude of adapter residual updates at individual spatial locations. This design encourages selective modifications to pretrained representations, potentially limiting unnecessary perturbations in regions that require little additional adaptation while retaining flexibility for atypical patterns. Unlike explicit regularization, this behavior is learned end-to-end through the task loss, without additional routing supervision or rarity labels.

To provide direct qualitative evidence of this behavior, Figure 9 in Appendix A.4 visualizes the learned spatial gates across feature stages C1–C3 in ISIC 2018 skin lesion classification task. The shallow gate (C1) exhibits relatively broad responses to low-level textures, whereas C2 shows more localized responses near lesion boundaries. At the deeper stage (C3), the relative gate responses become more concentrated around lesion regions, with generally weaker activation over the surrounding background. These stage-dependent patterns emerge without explicit spatial supervision, supporting the interpretation of gating as a learned mechanism for selective residual modulation.

We emphasize that these visualizations characterize the behavior of the learned gate and do not, by themselves, establish a causal improvement in predictive accuracy or training stability. Nevertheless, together with the multi-seed results, they help distinguish the two architectural roles within REAP: the bottleneck adapter provides additional adaptation capacity, while the spatial gate introduces input-conditioned control over how that capacity is expressed.

We appreciate the reviewer's concern and will clarify this distinction in the manuscript to avoid attributing the overall performance gains solely to the gating mechanism.

Reviewer_GxhbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes REAP, a residual adapter for improving rare-sample learning during post-training of hierarchical networks. Bottleneck transformations are modulated by input-dependent spatial gates, and a zero-initialized output projection initially preserves the pretrained function. REAP is combined with gradient-based importance sampling, while the encoder, decoder, and adapters are trained jointly. Experiments cover image classification, segmentation, remaining-useful-life prediction, and insurance severity regression using the SegFormer family.

Critical strengths
  1. The combined method shows useful gains on rare categories.On ADE20K, IS+REAP improves tail-category performance over both Base and IS while maintaining head-category performance. This provides concrete evidence of practical value on a benchmark where rarity is defined by class frequency.
  2. The adaptation scheme does not require additional routing labels. REAP (IS+REAP) is trained with the existing task loss, and its zero-initialized output projection preserves the pretrained model's function when the module is first inserted. This offers a practical way to add input-dependent adaptation to pretrained hierarchical models without collecting extra routing annotations.s show where the method works and where its benefits are limited.
Critical weaknesses
  1. The main architectural contribution is not sufficiently supported. Adding REAP improves several results, but there is no matched comparison with a similar adapter without input-dependent gates. The experiments therefore do not show whether the proposed gates provide a benefit beyond adding adaptation capacity. Since spatial gating is a central contribution, this missing comparison limits the evidence that the proposed design offers an advantage over a simpler alternative.
  2. The claimed rare-sample mechanism lacks direct validation. The paper argues that IS increases exposure to rare samples and REAP adapts to them. Yet large gradients do not necessarily indicate rarity. The paper does not directly show that sampling increases rare-sample exposure or that gates target rare patterns. ADE20K shows rare-category gains, but these alone do not verify the proposed mechanism, limiting support for the central claim of targeted rare-sample learning.
Questions for authors

Questions addressing the critical weaknesses

See weakness above.

Clarification requests about inconsistent descriptions (Not independent grounds for rejection)

  1. In Appendix A.2.2, Eq. (8) and the definition d=y−y^d = y − \hat{y} imply a penalty direction opposite to the text. Which definition was used to compute Table 6, and does the correction affect the reported scores or their interpretation?
  2. The insurance training loss is described as weighted log-squared error on p. 8 and as raw-scale RMSE on p. 16. Which loss was actually used to obtain Table 7, and is the discrepancy only in the description?
1 comment
AuthorsOpenReview ↗

Response to Gating, Rare-Sample Mechanism, and Implementation Clarifications

We thank the reviewer for the careful assessment and constructive suggestions. We have revised the manuscript to address both critical weaknesses and clarify the two inconsistencies.

1. Contribution of input-dependent gating.

We introduce a Plain Adapter baseline that retains the same residual bottleneck architecture but removes input-dependent spatial gating. Table 8 compares No Adapter, Plain Adapter, and REAP under identical sampling protocols. Under IS, REAP improves PHM08 on both FD001 and FD004 relative to the Plain Adapter, while also achieving the lowest RMSE on FD004. These results support the contribution of spatial gating beyond the shared residual adaptation capacity.

2. Validation of the rare-sample mechanism.

We introduce an instance-level analysis on 1,149 ISIC validation images (Figure 3), measuring visual rarity independently using an ungated baseline. Gate activation correlates positively with visual rarity (Spearman's ρ = 0.717), with the association persisting after within-class centering (ρ = 0.494). Figure 1(c) further reveals stage-dependent correction patterns, with shallow adapters responding more strongly to high-gradient samples.

Additionally, Section 5.2 and Appendix A.3 provide a theoretical analysis linking gradient norms to prediction difficulty under specified sensitivity assumptions. The analysis also derives an exact condition under which mixed IS increases the expected exposure of a sample subset. This clarifies the sampling mechanism without assuming that all rare samples necessarily have large gradients.

3. Clarification of the PHM08 scoring convention.

We have corrected the sign convention in Appendix A.2.2, Eq. (8), defining the prediction error as di=y^i−yid_i=\hat{y}_i-y_i. Accordingly, overestimating RUL (di>0d_i>0) incurs the stronger penalty, consistent with the description of delayed maintenance warnings. The revised formulation resolves the inconsistency between the equation and its interpretation.

4. Clarification of the freMTPL2 training objective.

The freMTPL2 severity model is optimized using claim-count-weighted squared error in log1p space, rather than raw-scale RMSE. We have clarified this consistently in Section 4.4 and Appendix A.2.1–A.2.2 (Eq. 10). Raw-scale RMSE is reported as an evaluation metric in Table 7, not used as the freMTPL2 training objective.

We hope these additional analyses and clarifications address the reviewer's concerns regarding the proposed mechanism, its empirical validation, and the consistency of the experimental descriptions. These corrections concern the manuscript descriptions only; the evaluation and training implementations remain unchanged, and no reported numerical results are affected.

Reviewer_Fa6jOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes REAP to address the underfitting of rare samples in deep learning models. REAP inserts residual adapters at multiple stages of a pretrained backbone and uses input-dependent gates to modulate feature corrections at each location. A zero-initialized output projection preserves the model’s behavior when the adapters are first introduced. The gates are learned end-to-end using only the task loss, without additional rarity labels or routing supervision. Experiments using SegFormer cover medical image, artificially constructed long-tailed tasks, semantic segmentation, engine remaining useful life prediction, and insurance claim severity regression. On ADE20K, REAP improves tail-class IoU by 10.64 percentage points over the base model, demonstrating its potential for long-tailed segmentation

Critical strengths
  1. The method only introduces a light-weight module onto a pre-trained model. This will not introduce significant overhead and will be practical for deployment.
  2. The paper shows accuracy gains over several tasks, showing the effectiveness of REAP.
  3. No additional signals are required.
Critical weaknesses
  1. The ablation study can be improved to isolate the contribution of the proposed gating mechanism. Since REAP introduces additional adapters, it remains unclear whether the improvements arise from increased model capacity and targeted training, or from the gating design itself. A comparison with ungated adapters under the same training protocol and parameter budget would help clarify this distinction.
  2. The motivation for the specific architectural design could be better polished. The experiments primarily demonstrate its empirical effectiveness, while providing limited evidence that it learns the intended rare-sample adaptation behavior. Beyond benchmark improvements, more qualitative and quantitative analyses, such as visualizations and statistics relating sample rarity, gate activations, and prediction quality, would strengthen the justification for the method.
Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Concerns on Gating Effectiveness and Rarity-Aware Adaptation

We thank the reviewer for the constructive feedback. We have expanded Section 5.1 with additional experiments and analyses addressing both concerns.

1. Isolating the contribution of input-dependent gating.

We introduce a Plain Adapter baseline that retains the same residual bottleneck architecture but removes input-dependent spatial gating. Table 8 compares No Adapter, Plain Adapter, and REAP under the same uniform and importance sampling protocols. Under IS, REAP improves PHM08 on both FD001 and FD004 relative to the Plain Adapter, while also achieving the lowest RMSE on FD004. These results provide evidence that the benefits of REAP extend beyond simply introducing additional adaptation capacity.

2. Validating rarity-aware adaptation behavior.

We further introduce an instance-level analysis on 1,149 ISIC validation images (Figure 3). Sample rarity is measured independently using nearest-neighbor distances in the feature space of an ungated baseline. Gate activation exhibits a strong positive association with visual rarity (Spearman's ρ = 0.717), which persists after controlling for diagnostic class (ρ = 0.494). Additionally, Figure 1(c) reveals distinct stage-dependent correction patterns: shallow adapters respond more strongly to high-Titlegradient samples, whereas deeper adapters exhibit more conservative responses.

Together, these analyses provide direct evidence of rarity-sensitive gating behavior and further support the contribution of input-dependent adaptation beyond the empirical performance improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.