acceptodds
Under review as a conference paper at ICLR 2027

Restricted Pyramid Graph Learning for Laryngopharyngeal Tumor Detection

Abstract

Accurate detection of laryngo-pharyngeal tumors from endoscopic images is critical for early diagnosis, yet remains challenging because lesions are often small, low-contrast, and visually similar to surrounding mucosa or imaging artifacts. Classifying laryngo-pharyngeal endoscopic images requires both local lesion detail and surrounding anatomical context. Lesion-guided methods provide a region of interest, but their encoders usually process subsequent feature maps without an explicit spatial relation between coarse and fine evidence. As a result, background responses re-enter during fine-scale refinement, contaminating attention and obscuring subtle lesion boundaries. To bridge this gap, we present RPGNet, which combines mask-supervised dual-view learning with coarse-to-fine feature routing. An adapted SAM 2 generates lesion-focused views and provides spatial supervision for attention learning, while a shared GreedyViG encoder extracts complementary global and local representations without feeding lesion masks to the classifier. A Restricted-Pyramid Network adaptively allocates the retained feature budget according to regional complexity, preserving informative details while suppressing redundant background. On the internal FAHSYSU test set, RPGNet reaches 94.85% accuracy and 92.55% macro-F1. On three held-out external cohorts, its macro-F1 exceeds the strongest result in the comparison table by 3.41, 3.38, and 9.58 percentage points on SAHSYSU, NHSMU, and FAHSU, respectively. These findings support spatially constrained multiscale selection while motivating further work on minority-class recognition. Code is available at https://anonymous.4open.science/r/Q7M4V9K2D8R6X3TP/.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Laryngopharyngeal Tumor Detection, Tree-pyramid Graph Learning,Mask-guided Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FGHeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RPGNet for three-class classification of laryngopharyngeal endoscopic images. It combines SAM2-guided dual-view learning, a shared GreedyViG encoder, and coarse-to-fine feature routing with a dynamic selection budget. It reports improved macro-F1 on one internal and three held-out external cohorts.

Critical strengths
  1. The paper evaluates RPGNet on one internal dataset and three external datasets, showing consistent improvements in macro-F1 over the compared methods. The multi-center evaluation provides encouraging evidence of the model's generalization ability.
  2. The proposed coarse-to-fine routing strategy is well motivated. By restricting feature selection through parent-child spatial relationships, the method offers a reasonable way to preserve lesion-related information across different scales.
  3. RPGNet achieves competitive performance with relatively low computational complexity. The reported parameter count and FLOPs are substantially lower than those of SAM-Swin, suggesting potential benefits for practical applications.
Critical weaknesses

Top-k selection. This makes it difficult to isolate the contribution of each component and justify the adaptive budget design. 2. The method largely builds on existing components, including SAM2, GreedyViG, and multiscale feature selection. Its distinction from existing hierarchical token selection and pruning methods is not sufficiently established. 3. The model still performs poorly on certain clinically important classes. Benign recall on FAHSU is only 27.29%, while malignant recall on NHSMU drops from 97.87% with SAM-Swin to 94.41% with RPGNet. 4. The claimed reduction in background interference lacks quantitative evidence. Since RPN operates on features already extracted by the backbone, it is unclear to what extent the proposed routing actually prevents background contamination. 5. RPGNet and the main competing methods use different backbones, making it difficult to attribute the performance gains to the proposed modules. Comparisons under matched backbone architectures and training settings would be more convincing.

Questions for authors
  1. Can the authors compare Dynamic-k with fixed-k and unrestricted Top-k selection under matched token budgets to better isolate the contribution of each component?
  2. How does RPN differ from existing hierarchical token selection methods? Can the authors provide quantitative evidence that it reduces background interference?
  3. Can the authors provide comparisons using the same backbone and training settings to better establish the contribution of the proposed modules?
  4. Could the authors further analyze the low benign recall on FAHSU and the reduced malignant recall on NHSMU compared with SAM-Swin?
10 comments
AuthorsOpenReview ↗

Evaluating Adaptive Budgets and Spatial Constraints

Weakness 1: Top-k selection. This makes it difficult to isolate the contribution of each component and justify the adaptive budget design.

Response:

We thank the reviewer for highlighting the need to clarify the contributions of the routing components. We provide two additional comparisons: fixed-k routing to examine the benefit of adaptive budgets, and unrestricted Top-k selection to evaluate the role of parent–child spatial constraints. All variants share the same encoder, supervision, Stage-2 training settings, and multiscale readout, allowing the comparisons to focus on the routing choices.

First, we retain the Scoring Module and parent–child spatial constraint while assigning a constant branching budget of k ∈ {0, 1, 2, 3, 4}. Each selected parent retains exactly k of its four candidate children, with their identities determined by the Scoring Module. These controls allow us to compare input-dependent allocation with constant branching within the same recursive routing framework.

Table 1. Comparison of fixed-k and Dynamic-k routing on FAHSYSU.

Budget settingACC (%)Macro-F1 (%)
093.6991.12
193.8991.54
294.2091.98
394.0891.83
493.9591.68
Dynamic (RPGNet)94.8592.55

Among the fixed-k controls, k = 2 gives the highest performance. Dynamic-k improves ACC from 94.20% to 94.85% and Macro-F1 from 91.98% to 92.55%, corresponding to gains of 0.65 and 0.57 percentage points, respectively. Its advantage over all five constant branching settings supports the use of adaptive allocation within the parent-constrained routing framework.

Second, we evaluate unrestricted Top-k selection, which allows child tokens to be selected regardless of whether their parent regions were selected. The multiscale readout remains unchanged, enabling us to evaluate the effect of the parent–child constraint while preserving the feature aggregation formulation.

Table 2. Unrestricted Top-k selection compared with full RPGNet across four datasets.

DatasetUnrestricted Top-k ACC (%)Full RPGNet ACC (%)Unrestricted Top-k Macro-F1 (%)Full RPGNet Macro-F1 (%)
FAHSYSU94.0594.8591.8992.55
SAHSYSU93.6994.4887.4588.68
NHSMU93.7694.7085.0587.21
FAHSU86.3887.8455.8156.35

Full RPGNet obtains higher ACC and Macro-F1 on every dataset. Relative to unrestricted selection, its Macro-F1 increases by 0.66 percentage points on FAHSYSU, 1.23 on SAHSYSU, 2.16 on NHSMU, and 0.54 on FAHSU. The improvements across both internal and external cohorts support the utility of preserving parent–child spatial correspondence during token selection.

Together, these experiments provide separate evidence for adaptive budget allocation and parent-constrained selection, with the multiscale readout held constant, and clarify the routing contributions beyond the original combined ablation.

AuthorsOpenReview ↗

Distinguishing RPN from Existing Token Selection and Pruning Methods

Weakness 2: The method largely builds on existing components, including SAM2, GreedyViG, and multiscale feature selection. Its distinction from existing hierarchical token selection and pruning methods is not sufficiently established.

Response:

We thank the reviewer for highlighting the need to clarify RPN's relationship to existing token selection and pruning methods. To address this concern, we explain the differences in where selection occurs, how candidate tokens are defined, and how retention budgets are assigned. RPGNet incorporates established components, including SAM2 and GreedyViG. The contribution we emphasize for RPN is the combination of spatially constrained parent–child selection and adaptive retention budgets predicted for individual parents within recursive multiscale routing.

For selection along network depth, DynamicViT [1] estimates token importance and progressively prunes less informative tokens at selected layers. A-ViT [2] uses token-wise halting to determine when individual tokens stop participating in subsequent layers. These methods regulate token participation during network computation. RPN operates on an already extracted feature pyramid and recursively selects tokens across spatial scales to construct multiscale descriptors for classification.

For multiscale token selection, MST [3] computes a reference feature by averaging base-token features associated with positive user clicks. It ranks multiscale tokens by their cosine similarity to this reference and applies Top-k selection with a specified retention count at each scale. The selected tokens then participate in feature interaction with the base tokens. RPN defines its candidate sets through explicit parent–child spatial correspondence and predicts the retention count separately for each retained parent.

Specifically, at each coarse-to-fine transition, a retained parent can select only from its four directly corresponding children at the next finer scale. The budget head uses the parent token to predict a retention count kₚ ∈ {0, 1, 2, 3, 4}, while the scoring head evaluates the child tokens and retains the top kₚ candidates within that parent's spatial footprint. The retained children become the parents for the next transition, and this procedure continues recursively across the feature pyramid.

RPN therefore couples where selection can occur, through the parent-constrained candidate set, with how many tokens to retain locally, through the parent-specific budget prediction. This formulation allows different parent regions within the same image to receive different branching budgets while preserving spatial correspondence across scales. The controlled comparisons reported in our response to Weakness 1 further evaluate the contributions of these two routing choices.

References

[1] Rao et al. “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.” NeurIPS, 2021.

[2] Yin et al. “A-ViT: Adaptive Tokens for Efficient Vision Transformer.” CVPR, 2022.

[3] L. Xu, Y. Chen, S. Li, and J. Luo, “MST: Adaptive Multi-Scale Tokens Guided Interactive Segmentation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 35, no. 8, pp. 8148–8160, 2025, doi: 10.1109/TCSVT.2025.3546634.

AuthorsOpenReview ↗

Class-wise Recall Limitations on External Cohorts

Weakness 3: The model still performs poorly on certain clinically important classes. Benign recall on FAHSU is only 27.29%, while malignant recall on NHSMU drops from 97.87% with SAM-Swin to 94.41% with RPGNet.

Response:

We thank the reviewer for drawing attention to these clinically important class-wise results. To better interpret them, we evaluate the class composition of the test cohorts and assess the reported recall values alongside the aggregate classification results.

Table 3. Class composition of the internal and external test cohorts.

DatasetNormal, n (%)Benign, n (%)Malignant, n (%)Total
FAHSYSU2,280 (31.54%)1,199 (16.59%)3,750 (51.87%)7,229
NHSMU5,716 (85.52%)592 (8.86%)376 (5.63%)6,684
FAHSU4,374 (89.96%)447 (9.19%)41 (0.84%)4,862

The external cohorts contain predominantly normal images. Benign recall on FAHSU is based on 447 images, while malignant recall on NHSMU is based on 376 images, compared with 1,199 benign and 3,750 malignant images in FAHSYSU. Each misclassification therefore has a larger effect on the external-cohort recall estimates. These counts provide context for interpreting the estimates and their sampling sensitivity; attributing the observed errors to class imbalance would require additional evidence.

We acknowledge that the benign recall of 27.29% on FAHSU indicates limited sensitivity for this class. On NHSMU, malignant recall decreases from 97.87% with SAM-Swin to 94.41% with RPGNet, a reduction of 3.46 percentage points. Both models are evaluated on the same cohort with the same class composition. Explaining this gap requires further analysis of model behavior, and the lower malignant recall remains a limitation of RPGNet in this comparison.

At the same time, RPGNet improves Macro-F1 over SAM-Swin by 3.38 percentage points on NHSMU and 9.58 percentage points on FAHSU. Since Macro-F1 assigns equal weight to each class, these gains provide complementary evidence of improved aggregate recognition across classes. The results therefore show stronger overall Macro-F1 on both external cohorts, together with persistent weaknesses in FAHSU benign recognition and lower NHSMU malignant sensitivity relative to SAM-Swin.

Reviewer_98L1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RPGNet for three-class classification of laryngopharyngeal endoscopic images into normal, benign, and malignant categories. SAM2-GLLM generates lesion-focused views, and a shared GreedyViG-B encoder extracts global and local features. A Restricted-Pyramid Network produces a multiscale representation through parent-constrained coarse-to-fine selection and dynamic retention budgets. This representation is fused with global and local descriptors for classification. Training follows two stages, with the encoder and view descriptors frozen during Stage 2. The paper reports classification results and component ablations on an internal test set and three external cohorts.

Critical strengths

1.The paper evaluates data from four centers, with three centers reserved for external testing. It explicitly states subject-disjoint internal splitting and evaluation of the comparators on the same split and external cohorts. RPGNet achieves macro-F1 scores of 88.68%, 87.21%, and 56.35% on SAHSYSU, NHSMU, and FAHSU, respectively, exceeding the strongest comparator in each cohort by 3.41, 3.38, and 9.58 percentage points. These results provide empirical support for its effectiveness on the evaluated external cohorts. 2.Table 3 provides component comparisons within the same dual-view framework. Relative to M4, which includes CDF and mask supervision, adding RPN with Dynamic-k improves macro-F1 by 1.49, 2.36, 3.28, and 6.17 percentage points across the four cohorts. This supports the usefulness of the combined routing addition, although the experiment does not separately establish the effects of spatial restriction and adaptive budgets.

Critical weaknesses

1.Table 3 supports the combined addition of RPN and Dynamic-k, but does not distinguish the contributions of parent-constrained selection, adaptive budgets, and the additional multiscale readout. Section 4.4 explicitly acknowledges the absence of a fixed-budget RPN and relevant standalone variants. Table 3 also lacks a matched control that aggregates the same multiscale features without parent constraints. Consequently, the reported improvements cannot be clearly attributed to the individual central design choices. This limits contribution attribution rather than demonstrating that the combined module is ineffective. 2.The main text and Algorithm 1 describe hard Top-k selection and the use of RankedSTE for surrogate gradients, but do not sufficiently define the specific surrogate gate, temperature setting, or gradient treatment through hard cross-level node indices. Appendix 6.1 and Table 4 primarily describe Stage 1, while the Stage-2 optimization configuration is not explicitly specified. Furthermore, the authors identify the results as single-run evaluations without quantifying variability across subjects or repeated training runs. The internal macro-F1 improvement over SAM-Swin is only 0.25 percentage points, leaving its stability unclear. These omissions limit reproducibility and assessment of the reliability of the gains.

Questions for authors

Q1.Please provide the specific forward and surrogate-backward definitions of RankedSTE in Algorithm 1, the value or schedule of temperature τ, and the gradient treatment of cross-level routing decisions after hard node indices are selected. Please also clarify whether the Stage-2 optimizer, learning rate, and learning-rate schedule follow Stage 1, and provide the actual configuration. Q2.To address the limitation acknowledged in Section 4.4, could you provide comparisons with a fixed-budget RPN and a multiscale readout without parent constraints under the same encoder, supervision, and Stage-2 training conditions, matching readout dimensionality and retention budgets where possible? Please also report the distributions of actual retained-token counts at each level to distinguish the effects of spatial restriction and adaptive budgets. Q3.Could you provide results from multiple random seeds and paired interval estimates for performance differences against the main comparators that account for correlation among images from the same subject? In particular, are the 0.25-percentage-point internal macro-F1 gain and the direction and magnitude of the malignant-recall decrease on NHSMU and benign-recall decrease on FAHSU stable across repeated runs?

7 comments
AuthorsOpenReview ↗

Contributions of Adaptive Budgets and Parent-Constrained Routing

Weakness 1: Table 3 supports the combined addition of RPN and Dynamic-k, but does not distinguish the contributions of parent-constrained selection, adaptive budgets, and the additional multiscale readout. Section 4.4 explicitly acknowledges the absence of a fixed-budget RPN and relevant standalone variants. Table 3 also lacks a matched control that aggregates the same multiscale features without parent constraints. Consequently, the reported improvements cannot be clearly attributed to the individual central design choices. This limits contribution attribution rather than demonstrating that the combined module is ineffective.

Response:

We appreciate the reviewer's observation that the combined ablation in Table 3 of the manuscript is insufficient to establish the contributions of the individual routing components. We further provide two controlled comparisons: fixed-k routing to evaluate adaptive budget allocation, and unrestricted Top-k selection to evaluate parent-constrained selection. Both comparisons retain the same encoder, supervision, Stage-2 training settings, and multiscale readout formulation as full RPGNet.

For the fixed-k controls, we preserve the Scoring Module and parent–child spatial constraints but replace Dynamic-k with k ∈ {0, 1, 2, 3, 4}. Each selected parent retains exactly k of its four candidate children, with their identities determined by the Scoring Module. These controls evaluate constant branching budgets within the same recursive routing framework.

Table 1. Comparison of fixed-k and Dynamic-k routing on FAHSYSU.

Budget settingACC (%)Macro-F1 (%)
093.6991.12
193.8991.54
294.2091.98
394.0891.83
493.9591.68
Dynamic (RPGNet)94.8592.55

The strongest fixed-k result is obtained at k = 2, with 94.20% ACC and 91.98% Macro-F1. Full RPGNet achieves 94.85% ACC and 92.55% Macro-F1, corresponding to improvements of 0.65 and 0.57 percentage points, respectively. These results support the benefit of adaptive allocation over the evaluated constant-budget settings.

We additionally evaluate unrestricted Top-k selection, which selects child tokens independently of whether their parent regions have been selected. The multiscale readout remains unchanged, enabling us to evaluate the parent–child spatial constraint within the same feature aggregation framework.

Table 2. Comparison between unrestricted Top-k selection and full RPGNet across four datasets.

DatasetUnrestricted Top-k ACC (%)Full RPGNet ACC (%)Unrestricted Top-k Macro-F1 (%)Full RPGNet Macro-F1 (%)
FAHSYSU94.0594.8591.8992.55
SAHSYSU93.6994.4887.4588.68
NHSMU93.7694.7085.0587.21
FAHSU86.3887.8455.8156.35

Full RPGNet achieves higher ACC and Macro-F1 on all four datasets. The Macro-F1 improvements over unrestricted selection are 0.66, 1.23, 2.16, and 0.54 percentage points on FAHSYSU, SAHSYSU, NHSMU, and FAHSU, respectively. These results support the utility of parent-constrained routing under the evaluated settings.

Together, these comparisons evaluate adaptive budget allocation and parent-constrained selection while holding the multiscale readout constant. They provide additional evidence beyond the combined ablation in the manuscript.

AuthorsOpenReview ↗

RankedSTE Gradients and Stage-2 Optimization

Weakness 2: The main text and Algorithm 1 describe hard Top-k selection and the use of RankedSTE for surrogate gradients, but do not sufficiently define the specific surrogate gate, temperature setting, or gradient treatment through hard cross-level node indices. Appendix 6.1 and Table 4 primarily describe Stage 1, while the Stage-2 optimization configuration is not explicitly specified. Furthermore, the authors identify the results as single-run evaluations without quantifying variability across subjects or repeated training runs. The internal macro-F1 improvement over SAM-Swin is only 0.25 percentage points, leaving its stability unclear. These omissions limit reproducibility and assessment of the reliability of the gains.

Response:

We thank the reviewer for identifying the missing implementation details and emphasizing the need to assess result stability. We specify the forward and surrogate-backward operations of RankedSTE, clarify gradient propagation through cross-level routing, and provide the Stage-2 optimization settings. The five-seed comparison with SAM-Swin on FAHSYSU is provided in Part 2.

At each feature level, let ss denote the candidate scores, Kˉ=E[K]\bar K=\mathbb{E}[K] the predicted continuous budget, and kk the integer budget used for hard selection. After ranking valid candidates in descending order of score, let πr\pi_r identify the candidate at rank rr. The hard selection gate and its continuous surrogate are defined as

gπrhard=1[r≤k],qπr=σ!(Kˉ−r+12τ).g^{\mathrm{hard}}_{\pi_r}=\mathbf{1}[r\leq k], \qquad q_{\pi_r}=\sigma\\!\left(\frac{\bar K-r+\tfrac12}{\tau}\right).

g~=ghard+q−sg⁡(q).\widetilde g=g^{\mathrm{hard}}+q-\operatorname{sg}(q).

where σ\sigma is the sigmoid function and sg⁡\operatorname{sg} is the stop-gradient operator. Both gates are set to zero for invalid candidates. The forward computation performs hard Top-k selection, while the backward computation uses the surrogate derivative

∂g~πr∂Kˉ=qπr(1−qπr)τ.\frac{\partial \widetilde g_{\pi_r}}{\partial \bar K}=\frac{q_{\pi_r}(1-q_{\pi_r})}{\tau}.

This path allows the classification loss to update the budget head through the continuous budget prediction.

The ranking operation itself is non-differentiable: we apply neither soft sorting nor a surrogate derivative to argsort. The scoring head instead receives gradients through the continuous score weights in the readout:

a=softmax⁡(s)⊙g~,a=\operatorname{softmax}(s)\odot\widetilde g,

with invalid candidates masked. The surrogate temperature is fixed at τ=0.5\tau=0.5 at every level throughout training, with no annealing. This temperature controls only the sigmoid surrogate gate and is not applied to the scoring softmax.

For cross-level routing, the indices returned by SelectedNodes and the support masks constructed by UnionSupport are treated as discrete, non-differentiable states. Consequently, finer-level losses do not update coarser-level scoring or budget heads through changes in these routing decisions. Gradients remain available through the gathered feature values. Each level also contributes its own score-weighted descriptor to the final concatenated representation, allowing the classification loss to supervise that level's scoring and budget heads through its descriptor path.

During Stage 2, the backbone is frozen, and the remaining trainable modules are optimized using AdamW. The learning rate starts at 1 × 10⁻⁵ and follows cosine decay to 1 × 10⁻⁶, without warm-up.

Table 3. Stage-2 optimization settings.

HyperparameterValue
OptimizerAdamW
BackboneFrozen
Initial learning rate for trainable modules1 × 10⁻⁵
Batch size512
Maximum epochs15
Input size256 × 256
Warm-upNone
Learning-rate scheduleCosine decay
Minimum learning rate1 × 10⁻⁶
Weight decay0.05
Label smoothing0
Default random seed42
AuthorsOpenReview ↗

RankedSTE Implementation and Stage-2 Training

Question 1: Please provide the specific forward and surrogate-backward definitions of RankedSTE in Algorithm 1, the value or schedule of temperature τ, and the gradient treatment of cross-level routing decisions after hard node indices are selected. Please also clarify whether the Stage-2 optimizer, learning rate, and learning-rate schedule follow Stage 1, and provide the actual configuration.

Response:

Please see our response to Weakness 2, Part 1, for the forward and surrogate-backward definitions of RankedSTE, and the gradient treatment through discrete cross-level routing. Table 3 provides the actual Stage-2 configuration, including the frozen backbone.

Reviewer_o2qHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RPGNet, a lesion-guided framework for laryngo-pharyngeal tumor detection from endoscopic images. The method combines SAM2-based lesion-focused views and mask-supervised attention learning with a shared GreedyViG encoder. Its core component, the Restricted-Pyramid Network, performs coarse-to-fine spatial feature selection through parent-constrained routing and an adaptive Dynamic-k budget. A Lesion-Aware Fusion module further integrates global, local, and restricted-pyramid representations for three-class classification. Experiments are conducted on one internal dataset and three external cohorts from four medical centers. The results show that RPGNet achieves competitive performance across these datasets, with particularly notable macro-F1 improvements on the external cohorts. However, the contribution of adaptive routing, the reliability of automatically generated lesion masks, and the robustness of minority-class recognition could be further investigated.

Critical strengths
  1. The paper presents a well-motivated approach to lesion-guided feature selection. Rather than relying only on lesion localization or conventional hierarchical feature extraction, RPGNet introduces parent-constrained coarse-to-fine routing to restrict which spatial features are retained at finer scales. This provides a reasonable way to reduce irrelevant background responses while preserving lesion-related details.
  2. The experimental evaluation covers multiple medical centers and includes three held-out external cohorts. RPGNet achieves consistent improvements in macro-F1 over the compared methods, particularly on SAHSYSU and NHSMU. The paper also provides component ablations, class-wise results, visualization, and computational complexity analysis.
Critical weaknesses
  1. The individual contribution of the proposed adaptive routing mechanism is not sufficiently isolated. The full RPGNet improves over M4 by adding RPN with Dynamic-k, but the ablation does not include fixed-budget routing or an RPN variant without Dynamic-k. Therefore, it remains unclear whether the improvements mainly come from spatially restricted parent-child selection or the adaptive feature budget. The claim that Dynamic-k allocates features according to regional complexity also lacks direct quantitative validation.
  2. The reliability of SAM2-generated lesion masks is not sufficiently evaluated. These masks provide the spatial supervision for the attention head and are also used to construct lesion-focused views. However, the paper does not provide quantitative comparisons with manually annotated lesion masks or analyze how segmentation errors affect the downstream classification results. This makes it difficult to assess the dependence of RPGNet on the quality of the automatically generated lesion regions.
  3. Minority-class recognition remains challenging on external datasets. Although RPGNet achieves strong aggregate macro-F1 results, its benign recall on FAHSU is only 27.29%, lower than SAM-Swin's 34.23%. Its malignant recall on NHSMU is also lower than that of SAM-Swin. In addition, the experiments report single-run image-level results without uncertainty estimates, making the stability of the reported improvements difficult to assess.
Questions for authors

1.Could the authors provide additional ablations comparing Dynamic-k with fixed-budget routing or other RPN variants? It would also be helpful to analyze whether the learned feature budgets are actually correlated with lesion complexity and contribute to the observed performance improvements. 2.Since SAM2-generated masks provide the spatial supervision for attention learning and are used to construct lesion-focused views, how do the authors verify their reliability? Could the authors provide comparisons with manually annotated masks or an analysis of how mask quality affects downstream classification? 3.Could the authors provide further analysis of the model's class-wise performance and its statistical stability across external cohorts? In particular, how does RPGNet address the poor benign recall on FAHSU, and can the authors report repeated-run results or confidence intervals to support the robustness of the improvements?

5 comments
AuthorsOpenReview ↗

Routing Ablations and Token Allocation by Lesion Size

Weakness 1: The individual contribution of the proposed adaptive routing mechanism is not sufficiently isolated. The full RPGNet improves over M4 by adding RPN with Dynamic-k, but the ablation does not include fixed-budget routing or an RPN variant without Dynamic-k. Therefore, it remains unclear whether the improvements mainly come from spatially restricted parent-child selection or the adaptive feature budget. The claim that Dynamic-k allocates features according to regional complexity also lacks direct quantitative validation.

Response:

We thank the reviewer for highlighting the need for clearer contribution attribution and quantitative validation of token allocation. We address these concerns through three complementary analyses: fixed-k routing to evaluate adaptive budget allocation, unrestricted Top-k selection to evaluate the parent–child spatial constraint, and lesion-size-stratified analysis to investigate the relationship between lesion extent and token allocation. The routing variants use the same encoder, supervision, Stage-2 training settings, and readout formulation.

To evaluate adaptive allocation against fixed routing, we compare Dynamic-k with five fixed-k controls (k ∈ {0, 1, 2, 3, 4}). In each control, every parent retains exactly k of its four candidate children at every routing stage, thereby fixing the number of branches at each parent; the Scoring Module still determines which children are retained. Comparing these controls with Dynamic-k assesses the contribution of adaptive per-parent budgets.

Table 1. Comparison of fixed-k and Dynamic-k routing on FAHSYSU.

Budget settingACC (%)Macro-F1 (%)
093.6991.12
193.8991.54
294.2091.98
394.0891.83
493.9591.68
Dynamic (RPGNet)94.8592.55

On FAHSYSU, the best fixed-k variant achieves a Macro-F1 of 91.98%, compared with 92.55% for full RPGNet, an improvement of 0.57 percentage points. These results support the benefit of adaptive budget allocation over the evaluated fixed-budget settings.

To examine the role of spatially constrained selection, we evaluate an unrestricted Top-k variant that removes the requirement that selected child tokens belong to selected parent regions. This comparison assesses the performance difference between unrestricted selection and the parent-constrained routing used in full RPGNet.

Table 2. Comparison between unrestricted Top-k selection and full RPGNet across four datasets.

DatasetUnrestricted Top-k ACC (%)Full RPGNet ACC (%)Unrestricted Top-k Macro-F1 (%)Full RPGNet Macro-F1 (%)
FAHSYSU94.0594.8591.8992.55
SAHSYSU93.6994.4887.4588.68
NHSMU93.7694.7085.0587.21
FAHSU86.3887.8455.8156.35

Full RPGNet achieves higher accuracy and Macro-F1 than unrestricted Top-k selection on all four datasets, with Macro-F1 improvements of 0.66, 1.23, 2.16, and 0.54 percentage points on FAHSYSU, SAHSYSU, NHSMU, and FAHSU, respectively. These results support the performance benefit of parent-constrained routing under the evaluated settings.

AuthorsOpenReview ↗

Routing Ablations and Token Allocation by Lesion Size

Weakness 1: The individual contribution of the proposed adaptive routing mechanism is not sufficiently isolated. The full RPGNet improves over M4 by adding RPN with Dynamic-k, but the ablation does not include fixed-budget routing or an RPN variant without Dynamic-k. Therefore, it remains unclear whether the improvements mainly come from spatially restricted parent-child selection or the adaptive feature budget. The claim that Dynamic-k allocates features according to regional complexity also lacks direct quantitative validation.

Response:

Our Dynamic-k mechanism adapts local token-retention budgets to the representation needs of different image regions. In this context, a relevant aspect of lesion complexity is spatial extent: larger lesions occupy more image regions and may require more tokens to preserve lesion coverage. We therefore focus on lesion area fraction as a measurable indicator of this spatial coverage requirement and evaluate whether token allocation increases with lesion extent.

Using SAM2-generated segmentation masks on FAHSYSU, we group images by lesion area fraction and calculate the average number of retained tokens at each feature level and in total.

Table 3. Average token counts across lesion-size groups on FAHSYSU.

Lesion area fraction (%)AVG No. tokens of F4AVG No. tokens of F3AVG No. tokens of F2AVG No. tokens of F1AVG total tokens
(0, 1]4.392.833.236.7317.18
(1, 5]4.924.938.6116.9435.40
(5, 10]8.728.4816.4132.2465.85
(10, 20]8.8512.4228.3054.42103.99
(20, 100]12.6918.7042.5282.69156.60

The average total number of tokens increases monotonically from 17.18 in the smallest-lesion group to 156.60 in the largest-lesion group. The average number of tokens at each feature level also increases across the lesion-size groups.

To quantify the relationship between lesion area and token allocation, we fit an unweighted linear regression using lesion-area interval midpoints and the average total number of retained tokens across the five groups, obtaining an R² of 0.8452.

The positive fitted slope indicates greater average token retention in larger lesion-area groups. Based on interval-midpoint approximations, this fit describes trends across group averages. Image-level associations and the relationship between lesion area and individual parent-specific budgets require separate evaluation.

These grouped results link greater lesion spatial extent with higher average token retention, consistent with allocation that responds to spatial coverage requirements. In this analysis, lesion area represents the spatial-extent component of complexity. The findings characterize trends across group averages; extending this interpretation to individual images or parent-specific budgets requires a separate analysis.

AuthorsOpenReview ↗

SAM2 Mask Reliability and Downstream Classification

Weakness 2: The reliability of SAM2-generated lesion masks is not sufficiently evaluated. These masks provide the spatial supervision for the attention head and are also used to construct lesion-focused views. However, the paper does not provide quantitative comparisons with manually annotated lesion masks or analyze how segmentation errors affect the downstream classification results. This makes it difficult to assess the dependence of RPGNet on the quality of the automatically generated lesion regions.

Response:

We thank the reviewer for highlighting the importance of mask reliability and its influence on downstream classification.

We fine-tune SAM2 using manually annotated images from FAHSYSU. The fine-tuned model then generates the lesion masks used in subsequent RPGNet training. These masks supervise the attention head and guide the construction of lesion-focused views. Table 4 reports the segmentation performance of the fine-tuned SAM2 model.

Table 4. Test-set segmentation performance of SAM2 fine-tuned on manually annotated FAHSYSU images.

Macro-Dice (%)Macro-IoU (%)Micro-Dice (%)Micro-IoU (%)
74.4363.3979.5165.99

To evaluate the downstream contribution of generated-mask supervision, we use the FAHSYSU ablation results in Table 3 of the manuscript. Adding the mask-attention loss (Lₘₐₛₖ) to M1 yields M3, while adding it to M2 yields M4. RPN remains disabled in both comparisons. Table 5 reports the performance changes relative to M1 and M2, respectively.

Table 5. Performance changes after adding Lₘₐₛₖ on FAHSYSU. All differences are in percentage points.

M1 with Lₘₐₛₖ (M3): ΔACCM1 with Lₘₐₛₖ (M3): ΔMacro-F1M2 with Lₘₐₛₖ (M4): ΔACCM2 with Lₘₐₛₖ (M4): ΔMacro-F1
+0.98+1.53+3.59+5.61

Adding mask-attention supervision improves both ACC and Macro-F1 on FAHSYSU in the two configurations. Combined with the segmentation evaluation, these results support the value of generated masks as supervision for internal-cohort classification. The comparisons assess this contribution at the observed mask quality; a controlled comparison across segmentation-error levels would quantify sensitivity to mask errors.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Routing Ablations and Lesion-Size Analysis

Question 1: Could the authors provide additional ablations comparing Dynamic-k with fixed-budget routing or other RPN variants? It would also be helpful to analyze whether the learned feature budgets are actually correlated with lesion complexity and contribute to the observed performance improvements.

Response:

Please see our response to Weakness 1, Part 1, for the fixed-k and unrestricted Top-k comparisons in Tables 1 and 2, which assess the contributions of adaptive budgets and spatially constrained routing. Weakness 1, Part 2, presents the lesion-size-stratified token allocation analysis in Table 3, using lesion area to capture the spatial-extent component of complexity.

AuthorsOpenReview ↗

SAM2 Mask Reliability and Downstream Classification

Question 2: Since SAM2-generated masks provide the spatial supervision for attention learning and are used to construct lesion-focused views, how do the authors verify their reliability? Could the authors provide comparisons with manually annotated masks or an analysis of how mask quality affects downstream classification?

Response:

Please see our response to Weakness 2 for the FAHSYSU annotation source and SAM2 fine-tuning workflow, the segmentation evaluation against manual annotations in Table 4, and the classification gains from mask-attention supervision in Table 5. These ablations assess the contribution of generated-mask supervision at the evaluated mask quality; sensitivity to segmentation errors would require controlled comparisons across mask-quality levels.

AuthorsOpenReview ↗

Class-wise Performance and Statistical Stability

Question 3: Could the authors provide further analysis of the model's class-wise performance and its statistical stability across external cohorts? In particular, how does RPGNet address the poor benign recall on FAHSU, and can the authors report repeated-run results or confidence intervals to support the robustness of the improvements?

Response:

Please see Weakness 3, Part 1, for the class-specific comparison with SAM-Swin and the cohort analysis, and Part 2 for the external repeated-run results in Table 7. Tables 8 and 9 provide the internal FAHSYSU matched-seed comparison, confidence intervals, and paired significance tests. The external results characterize RPGNet's performance variability, while the paired tests support its relative gains on the internal cohort.

Weakness 3: Minority-class recognition remains challenging on external datasets. Although RPGNet achieves strong aggregate macro-F1 results, its benign recall on FAHSU is only 27.29%, lower than SAM-Swin's 34.23%. Its malignant recall on NHSMU is also lower than that of SAM-Swin. In addition, the experiments report single-run image-level results without uncertainty estimates, making the stability of the reported improvements difficult to assess.

Response:

To assess performance stability, we report RPGNet's repeated-run results in Table 7 and compare it with SAM-Swin on FAHSYSU using five matched random seeds in Tables 8 and 9.

Table 7. RPGNet repeated-run performance summaries. Values are mean ± sample standard deviation (%).

DatasetACC (%)Macro-F1 (%)Normal recall (%)Benign recall (%)Malignant recall (%)
FAHSYSU94.69 ± 0.0892.34 ± 0.1697.21 ± 0.2378.92 ± 0.2398.18 ± 0.23
SAHSYSU94.21 ± 0.2588.50 ± 0.1696.00 ± 0.2479.74 ± 0.3293.71 ± 0.25
NHSMU94.44 ± 0.2487.03 ± 0.1696.43 ± 0.2475.41 ± 0.2794.15 ± 0.22
FAHSU87.37 ± 0.4356.17 ± 0.1693.66 ± 0.2424.65 ± 2.60100.00 ± 0.00

The FAHSYSU class-wise recall values follow the earlier summary; their correspondence to the five prediction sets used in Table 8 remains unverified.

Across the five external-cohort runs, ACC and Macro-F1 show relatively limited variation. Malignant recall on NHSMU is 94.15 ± 0.22%, whereas benign recall on FAHSU is 24.65 ± 2.60%, showing greater variability and persistently low sensitivity. Thus, low benign recall on FAHSU remains a limitation across the evaluated runs.

To assess the stability of the relative internal gain, we compare RPGNet and SAM-Swin on FAHSYSU using five matched random seeds. Table 8 summarizes their performance.

Table 8. Performance on FAHSYSU across five random seeds. Values are mean ± sample standard deviation (%).

MetricRPGNetSAM-Swin
ACC94.69 ± 0.0894.28 ± 0.20
Macro-F192.34 ± 0.1691.72 ± 0.20

We pair results by random seed and perform two-sided paired t-tests with five pairs. Table 9 reports the 95% confidence intervals for the mean paired differences and the corresponding two-sided p-values.

Table 9. Paired performance comparisons between RPGNet and SAM-Swin on FAHSYSU across five random seeds.

Metric95% CI of gain (percentage points)Two-sided p
ACC[0.2140, 0.5967]0.00417
Macro-F1[0.4115, 0.8443]0.00129

RPGNet achieves higher mean ACC and Macro-F1 than SAM-Swin. Both reported confidence intervals lie above zero, and both paired comparisons yield p < 0.05. These intervals characterize the mean paired differences across training seeds on the fixed FAHSYSU test cohort.

The paired tests provide statistical support for the internal FAHSYSU comparison. The external repeated-run results summarize RPGNet's performance variability; assessing the significance of its external differences from SAM-Swin requires paired repeated-run results from both models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.