acceptodds
Under review as a conference paper at ICLR 2027

One Query Is Enough: Adapting Frozen Multiple Instance Models

Abstract

Whole-slide image (WSI) classification increasingly relies on strong pretrained pathology encoders, yet the downstream multiple-instance learning (MIL) stage is still commonly adapted with a fully trainable attention network or transformer aggregator. This raises a basic question: how much task-specific adaptation is actually required once tile representations are strong? We study an intentionally minimal regime in which the tile encoder is frozen and the bag representation is adapted by a single learnable query. Given frozen tile embeddings, the query exponentially tilts their empirical distribution and forms a weighted expectation that is passed to a linear classifier. The representation-adaptation budget is therefore only one -dimensional vector beyond the task head. We show that this mechanism admits a precise characterization: query attention solves an entropy-regularized instance-selection problem, its representation Jacobian is a tile-feature covariance operator, and its temperature continuously interpolates between mean pooling and hard instance selection. The empirical study is designed around sufficiency rather than universal SOTA: the provisional result scaffold places one-query adaptation within one percentage point of a substantially richer full-data aggregator on fine-grained EBRAINS, while making its advantage larger in few-shot and external-cohort settings. Across three pathology encoders, the same scaffold predicts a narrow – point gap to the strongest in-domain aggregator, and the multi-query diagnostic saturates after one or two queries. These patterns motivate a broader hypothesis: with sufficiently strong pretrained pathology representations, much of downstream MIL adaptation may reside in how the bag is queried, rather than in relearning the representation itself.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Whole slide image

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_S4CUOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SQA-MIL, an MIL based approach for WSI classification using a frozen pathology foundation model. Instead of employing a complex trainable MIL aggregator, the method introduces a single learnable query to adaptively aggregate frozen patch representations.

Critical strengths

The paper propose a component that never been studied in the literature before, namely the query token of the PFM when we use it to extract the features.

From my view of point, the paper is interesting and well-motivated. And the paper shows that the even with a simple ABMIL, a better query model can improve model's performance even when comparing with more complicated MIL models.

Further, the paper might have potential implications for future PFM development, currently, all downstream tasks apply task agnostic query, in order to encoder richer information, researchers scaling the model size and the training data, I thought, the current method could inspire more efficient PFM architectures and downstream adaptation strategies.

The submission evaluate the method across different pathology encoders, limited-data regimes, and external validation. The results suggest that a single learnable query can achieve competitive performance with significantly fewer trainable parameters.

Critical weaknesses

My main concern is the lack of intuitive visualization showing how learned queries changes the representation space. As the central contribution of this paper is to propose a method which can improve the feature quality without scaling the PFM, I though a 2D visualization of the feature space can be useful, the author can consider to use TNSE or UMAP for the visualization.

Minor Comments

The submission title is inconsistent with the manuscript title. Please fix it.

Questions for authors

See the weakness.

3 comments
AuthorsOpenReview ↗

Respond to Reviewer S4CU

My main concern is the lack of intuitive visualization showing how learned queries changes the representation space. As the central contribution of this paper is to propose a method which can improve the feature quality without scaling the PFM, I though a 2D visualization of the feature space can be useful, the author can consider to use TNSE or UMAP for the visualization.

We thank the reviewer for suggesting an intuitive visualization of how the learned query adapts frozen pathology representations.

We have added t-SNE visualizations on EBRAINS and CAMELYON16 in Figure 1, comparing the original mean-pooled representations derived from frozen pathology features with the representations obtained after single-query adaptation. As shown in Figure 1, the learned-query representations exhibit substantially clearer class separation and more coherent class-wise clustering, whereas the original representations show considerable overlap between different classes. These observations provide an intuitive illustration of how a single learnable query changes the organization of slide-level representations to better support downstream classification.

Importantly, SQA-MIL does not modify the pretrained pathology foundation model or its tile embeddings. Instead, the learned query adaptively reweights the existing tile features to construct task-specific slide representations. We attribute the observed improvement to the rich discriminative information already encoded in the high-dimensional pretrained feature space: rather than learning new features, a single query learns how to selectively aggregate the existing information into a more discriminative slide-level representation.

This observation is further supported by quantitative measurements in the original high-dimensional representation space (Table B.9). On EBRAINS-fine, single-query adaptation improves linear-probe balanced accuracy from 0.620 to 0.716, increases the mean classification margin from 0.384 to 0.587, and raises the between-class/within-class scatter ratio from 0.041 to 0.058. These measurements confirm improved class discriminability in the original representation space, independently of the two-dimensional t-SNE projections.

Together, the visualizations and quantitative results reinforce our central finding: strong pretrained pathology features already contain substantial task-relevant information, and a single learnable query can effectively extract and aggregate this information to produce more discriminative slide representations without modifying or scaling the foundation model. The t-SNE visualizations provide an intuitive illustration of this mechanism, while the original-space measurements establish its quantitative effect.

The submission title is inconsistent with the manuscript title. Please fix it.

We thank the reviewer for pointing out this inconsistency. This was a submission-stage oversight: we inadvertently used an earlier working title in the submission form before finalizing the manuscript title. We have corrected the submission title to match the manuscript.

Thank you for the detailed response and for adding both the t-SNE visualizations and the quantitative representation-space analysis. My concerns have been fully addressed.

Reviewer S4CU

Dear Reviewer S4CU,

We sincerely appreciate your encouraging assessment and supportive rating. We are especially grateful for your recognition of the significance of task-specific query adaptation, its potential implications for future pathology foundation models, and its ability to achieve competitive performance without increasing foundation-model complexity.

We are delighted that the additional t-SNE visualizations and quantitative representation-space analysis have fully addressed your concerns. Your insightful suggestions have substantially strengthened the presentation of our central contribution. Thank you again for your thoughtful feedback and valuable support.

Best regards,

The Authors

Reviewer_ZickOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper provides the method SQA-MIL which is an approach to adapt frozen pathology foundation models to whole-slide image classification. The method develops a method to use a single shared learnable query assigns softmax attention weights to tile features, and their weighted average is passed to a linear classifier. The study provides an analysis of the operator through exponential-family and covariance identities and evaluate it across training-set sizes, pathology encoders, and external-cohort testing.

Critical strengths
  1. The study provides a simple and lightweight computational method for whole slide images (WSI) classification.

  2. The theoretical analysis provides an accessible interpretation of query pooling, convex-hull constraint, and class-shared attention distribution.

Critical weaknesses
  1. Learned-query pooling already provided by previous studies. ViLa-MIL[1] uses learnable prototypes as queries over frozen WSI patch features. This approach is also widely used by other studies such as MGPATH [2], MSPCT [3], and FOCUS[4].

  2. The study is lacking a complete coverage of relevant few-shot WSI methods. The paper includes ABMIL, DSMIL, TransMIL, ILRA, but lacking the most SOTA methods for few-shot lightweight computational WSI including ViLA-MIL, MGPATH, MSCPT and FOCUS. The studies address the questions involving adaptation, prompts, and aggregation of WSI features. SQA-MIL must explain its reltationship to these approaches and provide an extensive comparison experiments. Moreover, the few-shot setting is considered the standard for lightweight computational WSI. Therefore, the study should provide experiments in a few-shot setting with 5 repetitions.

[1] Shi et al. ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification. CVPR, 2024.

[2] Nguyen et al. MGPATH: A Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot Whole Slide Pathology Classification. TMLR, 2025.

[3] Han et al. MSCPT: Few-Shot Whole Slide Image Classification With Multi-Scale and Context-Focused Prompt Tuning. IEEE Transactions on Medical Imaging, 2025.

[4] Guo et al. FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification. CVPR, 2025.

Questions for authors
  1. Can you provide the five-seed variability for all the main experiments.

  2. Can you provide experiments to compare SQA-MIL with all SOTA methods such as MGPATH, ViLA-MIL, MSPCT, FOCUS in the setting of 16, 8, and 4 shots.

  3. What are the results of the stated class-specific-query control? How broadly do the available query-count results support the one-query claim across the evaluated encoders and settings?

7 comments
AuthorsOpenReview ↗

Respond to Reviewer Zick

We thank the reviewer for the detailed assessment and for highlighting the importance of positioning SQA-MIL relative to existing query-based and few-shot WSI classification methods. We appreciate the recognition of the method's computational simplicity and its analytical interpretation. The concerns regarding novelty, comparisons with recent few-shot methods, and experimental robustness are central to evaluating our contribution. We address these questions individually below, with particular attention to methodological distinctions and controlled experimental comparisons.

Learned-query pooling already provided by previous studies. ViLa-MIL[1] uses learnable prototypes as queries over frozen WSI patch features. This approach is also widely used by other studies such as MGPATH [2], MSPCT [3], and FOCUS[4].

We thank the reviewer for highlighting these relevant studies. We acknowledge that learnable-query attention has been explored in prior work. However, our approach differs fundamentally in the role of the learned vectors, architectural formulation, and research objective. First, prototype-based aggregation and our query-only adaptation serve distinct purposes. ViLa-MIL [1] employs multiple learnable prototypes as representative anchors to group and progressively aggregate semantically related patch features through cross-attention. In contrast, our single learnable query is not intended to represent morphological prototypes or construct a prototype-based representation. It acts exclusively as a task-dependent readout direction, assigning attention weights directly to unchanged frozen tile embeddings. MGPATH [2], MSPCT [3], and FOCUS [4] likewise introduce additional mechanisms involving vision-language prompting, contextual modeling, or knowledge-guided visual compression, rather than isolating a single query as the sole representation-adaptation component. Second, our methodological objective is fundamentally different. While these studies focus on improving representational expressiveness through richer aggregation or additional semantic guidance, SQA-MIL investigates the minimum adaptation capacity necessary to extract task-relevant information from already pretrained representations. Our formulation uses exactly one shared d-dimensional query, without prototype banks, trainable key/value projections, feature transformations, or auxiliary alignment objectives. The query is initialized at zero, exactly recovering mean pooling before learning a task-specific instance-selection distribution. Finally, our contribution lies in establishing and evaluating this minimal adaptation regime, rather than proposing another query-based aggregation mechanism. Through theoretical characterization and controlled experiments across full-data, few-shot, cross-encoder, and external-cohort settings, we quantify how much predictive performance can be retained with only d representation-adaptation parameters. On 30-class EBRAINS-fine, SQA-MIL achieves 0.721 balanced accuracy versus 0.727 for DGR-MIL, using 1,024 versus approximately 4.73 million representation-adaptation parameters. Therefore, although these methods share certain attention primitives, their use of prototypes, prompts, or richer aggregation structures is conceptually and methodologically distinct from our investigation of single-query sufficiency under strictly frozen representations.

Can you provide the five-seed variability for all the main experiments.

We thank the reviewer for this suggestion. We have now included the five-seed variability (mean and standard deviation) for all main experiments in the revised manuscript, providing a more transparent assessment of the robustness and reproducibility of our results.

The study is lacking a complete coverage of relevant few-shot WSI methods. The paper includes ABMIL, DSMIL, TransMIL, ILRA, but lacking the most SOTA methods for few-shot lightweight computational WSI including ViLA-MIL, MGPATH, MSPCT and FOCUS. The studies address the questions involving adaptation, prompts, and aggregation of WSI features. SQA-MIL must explain its reltationship to these approaches and provide an extensive comparison experiments. Moreover, the few-shot setting is considered the standard for lightweight computational WSI. Therefore, the study should provide experiments in a few-shot setting with 5 repetitions. Can you provide experiments to compare SQA-MIL with all SOTA methods such as MGPATH, ViLA-MIL, MSPCT, FOCUS in the setting of 16, 8, and 4 shots.

We thank the reviewer for highlighting these relevant few-shot WSI methods. In response, we have expanded our comparison to include ViLA-MIL, MGPATH, MSPCT, and FOCUS under the same experimental protocol as SQA-MIL. Specifically, we use frozen UNI features for visual representations and CONCH as the language encoder for the vision–language baselines. All methods are evaluated on EBRAINS-fine using 1-, 2-, 5-, 10-, and 15-shot settings, with five independent repetitions, reporting mean ± standard deviation of balanced accuracy. The expanded comparison examines different adaptation strategies, including prototype-based learning, language-guided prompting, and trainable feature aggregation. In contrast, SQA-MIL directly adapts the readout of frozen tile representations through a single learnable query, without additional feature transformations or trainable key/value projections. The additional experiments strengthen the evaluation of its predictive performance and parameter efficiency in low-data regimes. We have incorporated these comparisons into Table 2 and expanded the discussion in the revised manuscript.

What are the results of the stated class-specific-query control? How broadly do the available query-count results support the one-query claim across the evaluated encoders and settings?

We thank the reviewer for this important question. We have reported additional experimental evidence in Section 5.6 and Appendix Table B.6 to directly examine whether increasing query capacity improves classification performance. On the 30-class EBRAINS-fine task, introducing independent class-specific queries achieves a balanced accuracy of 0.722, compared with 0.721 for our single shared query, representing a marginal improvement of only 0.001 95% CI: [-0.07, 0.06]. Notably, this requires increasing the representation-adaptation parameters from 1,024 to 30,720 30×30\times. Consistently, the query-count ablation in Table B.6 reports balanced accuracies of 0.718, 0.724, 0.726, and 0.724 for 1, 2, 4, and 8 queries, respectively, demonstrating rapidly diminishing returns as query capacity increases. Beyond this controlled ablation, Tables B.10–B.11 provide complementary evidence across three pathology foundation encoders (CONCH, UNI, and Virchow). SQA-MIL consistently approaches DGR-MIL on EBRAINS-fine, with balanced-accuracy differences of only -0.009, -0.006, and -0.005, respectively, while outperforming DGR-MIL across all three encoders on external CPTAC validation. Taken together, these results demonstrate that increasing query capacity provides limited additional predictive benefit in the evaluated ablations, while single-query adaptation consistently retains competitive performance across encoders and evaluation settings. This supports our central argument that effective downstream adaptation does not necessarily require a more expressive aggregation architecture when strong frozen representations are available.

Thank you for the clarification and additional experimental results. I appreciate the inclusion of recent few-shot methods and the reporting of five-seed variability in the main comparisons. However, my concern regarding the significance and novelty of the contribution remains.

I acknowledge that SQA-MIL differs from the cited methods in its parameterization, architectural complexity, and research objective. However, these differences do not, by themselves, establish a fundamentally different aggregation mechanism. Previous approaches, such as ViLa-MIL, already employ learned vectors to aggregate instance features through attention.

I agree that demonstrating the effectiveness of a substantially simpler model can constitute a valuable contribution. My concern is whether the study is classified as an incremental improvement over the previous studies.

Therefore, I remain unconvinced that the methodological simplification and the present empirical characterization together establish a sufficiently substantial advance over existing query-based aggregation approaches.

Reviewer_q2RqOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

It proposed a simple attention-kind calculation based on single query vector with a limited dim projected by a Softmax using justified temperature to simplify the parameter selection to a controllable form. They propose an objective function using the entropy concept to encourage the alignment of instance representations with the query, while at the same time overfitting on limited samples is penalized. They show that their formulation limits the distribution of generated weights while maintaining the estimation power of learning perfect representation weights as a free attention mechanism.

Critical strengths
  • An interesting perspective.
  • The manuscript is well organized and easy to follow. Clear research question supported by a clear scientific writing style.
  • Good coverage of diverse and relevant related work.
  • Potentially applicable to multimodal and vision–language settings.
  • The approach is permutation-invariant (formally proved).
  • Limitations are mentioned at different levels, which are scientifically valuable in terms of scientific transparency.
Critical weaknesses
  1. The main findings and take-home message are missing from the abstract. The objective is stated as “how much performance is retained per representation-adaptation parameter”, but it is unclear what practical or scientific benefit this parameter efficiency provides. Does it improve model generalization, efficiency, interpretation, or what? Hint: the last sentence of Section 2 should be mentioned earlier. Relevant demonstrations (e.g., Table B.4) should be included in the main manuscript.
  2. Insufficient referencing of the corresponding experiments: Section 3 mentions several experimental validations without clearly indicating where their results are reported.
  3. The low-data experimental protocol is insufficiently detailed.
  4. The “best rich MIL” mentioned in the tables should be specified somewhere in the text or tables.
  5. Tables in the appendix contain important results, but are not appropriately referenced in the main text.
  6. The source code was not accessible to me, although the reproducibility section states that a source package file is provided!
Questions for authors
  1. What temperature was selected for each experiment, and how do you explain its effect on attention, concentration, and performance? A comparison across datasets with different (class-relevant) instance distributions (e.g., BRACS vs CAMELYON) could provide useful insights.
  2. Is there any relationship between the covariance and classification error? If not, what insights does the covariance form of the gradient provide beyond characterizing the optimization dynamics?
  3. Did you investigate the eigenvalue spectrum of attention-weighted covariance matrices across datasets, particularly in low-data settings? When tile embeddings within a bag are nearly identical, the covariance approaches zero, potentially causing vanishing query gradients. This seems worth examining as an important diagnostic of query learnability.
  4. In the multi-query setting, averaging the query readouts may result in the loss of class-specific information. Why did you average the readouts instead of simply using each class-specific query for the corresponding classifier head?
  5. And please refer to the weaknesses.
9 comments
AuthorsOpenReview ↗

Respond to Reviewer q2Rq

We thank the reviewer for the positive and constructive feedback. We appreciate the recognition of our clear research motivation, theoretical analysis, comprehensive related-work coverage, and discussion of limitations. The suggestions regarding the practical significance of parameter efficiency, experimental reproducibility, temperature sensitivity, and covariance analysis are valuable for strengthening the presentation and empirical understanding of SQA-MIL. We address each point below.

The main findings and take-home message are missing from the abstract. The objective is stated as “how much performance is retained per representation-adaptation parameter”, but it is unclear what practical or scientific benefit this parameter efficiency provides. Does it improve model generalization, efficiency, interpretation, or what? Hint: the last sentence of Section 2 should be mentioned earlier. Relevant demonstrations (e.g., Table B.4) should be included in the main manuscript.

We thank the reviewer for this constructive suggestion. We have revised the Abstract to explicitly highlight our main findings and practical contributions, including competitive predictive performance, improved low-data and external-cohort results, and substantially reduced downstream computational costs.

Specifically, SQA-MIL achieves 0.721 versus 0.727 balanced accuracy on EBRAINS-fine while using approximately 4,600 fewer representation-adaptation parameters than DGR-MIL, with 3.3×3.3 \times faster downstream training per slide and 71% lower peak GPU memory.

We have also revised the Introduction to clarify the scientific significance of constrained readout adaptation and moved the efficiency comparison from Appendix Table B.4 into the main Results section.

The updated paper highlight the finding, a single learnable query retains competitive predictive performance while substantially reducing the adaptation capacity and computational cost required to leverage frozen pathology representations.

Insufficient referencing of the corresponding experiments: Section 3 mentions several experimental validations without clearly indicating where their results are reported.

We thank the reviewer for highlighting this presentation issue. We agree that the appendix results should be more explicitly connected to the main text.

We have revised Section 5.6 to provide direct references to all 11 appendix tables, covering encoder configurations, optimization protocols, parameter-efficient baselines, computational efficiency, effective-batch controls, query-count ablations, temperature and initialization studies, representation discriminability, and cross-encoder robustness.

These revisions improve the accessibility and traceability of the experimental evidence, making the connection between our main claims and supporting results more explicit. All experimental results remain unchanged.

The low-data experimental protocol is insufficiently detailed.

We thank the reviewer for highlighting the need for greater experimental clarity. We have expanded Section 5.4 and Appendix B.1–B.3 to specify the low-data evaluation protocol.

The “best rich MIL” mentioned in the tables should be specified somewhere in the text or tables.

We thank the reviewer for pointing this out. We have clarified in the revised manuscript that “Best rich MIL” refers to DGR-MIL, the strongest-performing rich MIL baseline in our matched full-data EBRAINS-fine evaluation, and updated the table descriptions accordingly.

Tables in the appendix contain important results, but are not appropriately referenced in the main text.

Thank you for pointing this out. We have revised the main manuscript to explicitly reference the relevant appendix tables at the appropriate locations, ensuring that the supplementary experimental results are clearly connected to the corresponding discussions in the main text.

The source code was not accessible to me, although the reproducibility section states that a source package file is provided!

We thank the reviewer for raising this concern. We are currently organizing and documenting the code repository to facilitate its release. The decision regarding public code availability will be made based on the paper's acceptance outcome. We have also updated the Reproducibility Statement (Section 9) to clarify the current status of the source code and avoid any confusion regarding its availability.

What temperature was selected for each experiment, and how do you explain its effect on attention, concentration, and performance? A comparison across datasets with different (class-relevant) instance distributions (e.g., BRACS vs CAMELYON) could provide useful insights.

We thank the reviewer for this constructive suggestion. The temperature selection protocol is specified in Appendix B.3, where τ\tau is selected from {0.05, 0.10, 0.20, 0.50, 1.00} using validation balanced accuracy and remains fixed during training.

For EBRAINS-fine with UNI features, the best reported performance is achieved at τ=0.20\tau=0.20 (balanced accuracy: 0.721). As shown in Table B.7, increasing the temperature from 0.05 to 1.00 increases normalized attention entropy from 0.286 to 0.901, while classification performance peaks at an intermediate temperature. This demonstrates that moderate attention concentration balances selective instance weighting with the preservation of distributed morphological information. We have clarified the selected temperature in Appendix B.3.

Is there any relationship between the covariance and classification error? If not, what insights does the covariance form of the gradient provide beyond characterizing the optimization dynamics?

We thank the reviewer for this question. The covariance identity does not establish a direct relationship with classification error. Instead, it provides two insights beyond optimization dynamics: (i) it identifies feature directions associated with larger target-versus-predicted class margins, explaining how the query selectively reweights task-relevant instances; and (ii) its null space characterizes the local expressive limitations of query adaptation. Empirically, Appendix Table B.9 shows that SQA-MIL improves linear-probe balanced accuracy from 0.620 to 0.716 and mean classification margin from 0.384 to 0.587. These findings support improved representation discriminability without claiming a theoretical guarantee on classification error.

Did you investigate the eigenvalue spectrum of attention-weighted covariance matrices across datasets, particularly in low-data settings? When tile embeddings within a bag are nearly identical, the covariance approaches zero, potentially causing vanishing query gradients. This seems worth examining as an important diagnostic of query learnability.

We thank the reviewer for this insightful suggestion. Our theoretical analysis in Section 4.2 already establishes a direct connection between attention-weighted covariance and query optimization. Specifically, Theorem 1 shows that the representation Jacobian is proportional to the covariance matrix, while Corollary 1 characterizes the query update through the covariance between tile embeddings and class-discriminative margins. Importantly, low-data supervision does not necessarily imply low covariance, since the covariance is determined by the within-bag feature distribution and attention weights rather than the number of labeled training slides. At initialization, the query is zero and attention is uniform, so the covariance of a given bag remains unchanged across supervision budgets. Furthermore, a concentrated eigenvalue spectrum does not necessarily imply vanishing query gradients. Query adaptation remains effective along eigen-directions with nonzero eigenvalues, depending on their alignment with the classification objective. These theoretical properties provide a direct characterization of query learnability without requiring an additional spectral decomposition to establish the underlying mechanism. We agree that a dataset-level spectral analysis would provide complementary empirical insights, but it is distinct from the central objective of this work, which is to evaluate the predictive effectiveness and parameter efficiency of single-query adaptation across datasets and supervision budgets.

In the multi-query setting, averaging the query readouts may result in the loss of class-specific information. Why did you average the readouts instead of simply using each class-specific query for the corresponding classifier head?

We thank the reviewer for raising this important distinction.

The multi-query averaging design is intentional: its purpose is to isolate the effect of increasing query capacity while keeping the classifier architecture fixed. This ensures that performance differences arise from additional learnable queries rather than changes to class-specific routing.

We agree that averaging does not explicitly preserve independent class-specific readouts. However, this is a separate architectural question, which we address through the class-specific-query control already defined in Appendix B.4 (Eqs. 46–47). In this formulation, each class uses its own query and corresponding classifier weights without averaging.

The experimental results support our central finding that a single shared query captures most of the predictive benefit of class-conditioned readout adaptation in the evaluated setting.

Reviewer_ZrrDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates how much task-specific adaptation is necessary for whole-slide image classification when pretrained tile representations are frozen. The authors propose Single-Query Adaptation MIL (SQA-MIL), which uses a single learnable query to assign softmax attention weights to fixed tile embeddings and aggregates them into a slide-level representation for a linear classifier. The approach introduces only one d-dimensional vector for representation adaptation beyond the classifier parameters. The paper also provides a mathematical characterization of the mechanism through entropy-regularized exponential tilting, the covariance structure of the representation Jacobian, and the limiting behavior between mean pooling and instance selection. Experiments on EBRAINS and TCGA-to-CPTAC evaluation compare SQA-MIL with conventional MIL aggregators and parameter-efficient alternatives under full-data, few-shot, and cross-encoder settings. The results suggest that a highly constrained attention readout can retain much of the performance of more complex MIL models while requiring substantially fewer trainable representation parameters.

Critical strengths
  1. The paper investigates a meaningful question in computational pathology: whether strong frozen tile representations require complex downstream aggregation models for effective task adaptation. SQA-MIL offers a deliberately constrained solution that modifies the bag representation through only one learnable query while retaining a standard linear classifier. The method is computationally lightweight and its optimization behavior is clearly characterized through exponential tilting, covariance-based gradients, and temperature-dependent instance selection. Although these mathematical properties are largely inherited from standard attention mechanisms, their application to a strictly controlled adaptation setting provides an interpretable framework for studying the relationship between representation capacity and downstream performance.

  2. The paper evaluates SQA-MIL against conventional MIL aggregators and parameter-efficient alternatives using matched frozen features and dataset splits. The experiments cover several pathology foundation-model encoders, few-shot training conditions, and an external TCGA-to-CPTAC evaluation. In particular, SQA-MIL improves substantially over mean pooling and parameter-matched sigmoid and rank-one controls, while remaining competitive with more complex MIL aggregators. The reported advantages in low-data settings and external validation are especially encouraging, suggesting that restricting downstream adaptation capacity may provide practical benefits when pretrained instance representations are sufficiently informative.

Critical weaknesses
  1. The proposed method is essentially a highly constrained form of standard softmax attention pooling, in which a single learnable vector assigns attention weights to frozen instance embeddings. Similar query-based aggregation mechanisms have already been explored in MIL methods such as DGR-MIL, although with substantially more expressive architectures. The theoretical results concerning entropy-regularized reweighting, covariance-based gradients, and temperature limits are standard properties of exponential-family attention rather than fundamentally new mathematical insights. While the authors appropriately acknowledge these relationships and position the work as a controlled empirical investigation, the paper does not fully establish whether the single-query restriction provides a sufficiently substantial methodological contribution beyond simplifying existing attention pooling. Stronger evidence regarding the distinctive benefits and limitations of this constrained formulation would improve its significance.

  2. The statistical evidence and evaluation of the single-query bottleneck are not sufficiently complete. Although the paper reports encouraging performance, several comparisons involve relatively small differences. For example, on full-data EBRAINS-fine, SQA-MIL achieves 0.721 balanced accuracy compared with 0.727 for the strongest rich MIL baseline, while its external CPTAC improvement over DGR-MIL is 0.931 versus 0.914. Appendix B.6 describes five-seed variability and paired bootstrap confidence intervals, but the displayed result tables provide only point estimates, making it difficult to assess the reliability of these differences. Furthermore, although a class-specific-query control is introduced to examine the limitations of using a shared attention distribution across classes, its quantitative results are not reported. Since the central hypothesis concerns whether one query is sufficient for downstream adaptation, more complete statistical reporting and a direct evaluation of this expressive bottleneck are necessary to substantiate the main conclusions.

Questions for authors
  1. How statistically robust are the reported improvements over stronger MIL baselines? The paper describes evaluations over five matched training seeds and paired bootstrap comparisons, but the main tables primarily report point estimates without corresponding standard deviations or confidence intervals. Could the authors provide seed-level variability and paired 95% confidence intervals for the principal comparisons, particularly SQA-MIL versus DGR-MIL on EBRAINS-fine and external CPTAC? Since several performance differences are relatively small, this information would materially affect my assessment of the empirical evidence.

  2. How restrictive is the class-shared single-query representation in fine-grained classification? The proposed formulation assigns the same attention distribution to all classes, which may limit its ability to identify distinct morphological evidence for different diagnoses. Although the paper introduces a class-specific-query control, its quantitative results are not clearly reported. Could the authors provide this comparison, particularly on the 30-class EBRAINS-fine task, and analyze whether different classes benefit from independent attention distributions? Such evidence would help establish whether the observed performance reflects the sufficiency of a single shared query or whether additional task-specific selection capacity is necessary.

  3. How broadly does the single-query advantage generalize beyond the evaluated classification settings? The cross-encoder results are encouraging, but the external-cohort experiment focuses on a binary LUAD-versus-LUSC classification task, whereas the more challenging fine-grained evaluation is conducted within EBRAINS. Could the authors clarify the expected limitations of the method for heterogeneous multi-class classification under cohort shift, and, if feasible, provide an additional external or cross-cohort multi-class evaluation? This would help determine whether the reported generalization advantage is specific to the evaluated binary task or represents a broader property of constrained readout adaptation.

7 comments
AuthorsOpenReview ↗

Respond to Reviewer ZrrD

The proposed method is essentially a highly constrained form of standard softmax attention pooling, in which a single learnable vector assigns attention weights to frozen instance embeddings. Similar query-based aggregation mechanisms have already been explored in MIL methods such as DGR-MIL, although with substantially more expressive architectures.

The reviewer identifies an important distinction between architectural novelty and the scientific contribution of the proposed adaptation regime. We do not claim that our contribution is a new attention mechanism. SQA-MIL is deliberately a restricted member of the attention-pooling family. The central difference is the capacity constraint imposed on task-specific representation adaptation: SQA-MIL learns only q∈Rdq\in\mathbb{R}^d to modify the frozen bag representation. It introduces no trainable key/value projections, instance-level feature transformations, or auxiliary aggregation networks. Further, the pretrained tile embeddings remain unchanged, and the classifier is a linear map applied after aggregation. Our controlled experiments isolate the contribution of task-dependent instance reweighting from the contribution of larger trainable aggregation networks. In the existing UNI experiments, SQA-MIL reaches 0.721 balanced accuracy on the 30-class EBRAINS-fine task, compared with 0.727 for DGR-MIL, while using 1,024 rather than approximately 4.73 million trainable representation parameters. We have revised the introduction and related work to state explicitly that the proposed contribution concerns the sufficiency of a minimal attention parameterization, not the introduction of a new attention primitive.

The theoretical results concerning entropy-regularized reweighting, covariance-based gradients, and temperature limits are standard properties of exponential-family attention rather than fundamentally new mathematical insights. While the authors appropriately acknowledge these relationships and position the work as a controlled empirical investigation, the paper does not fully establish whether the single-query restriction provides a sufficiently substantial methodological contribution beyond simplifying existing attention pooling. Stronger evidence regarding the distinctive benefits and limitations of this constrained formulation would improve its significance.

We thank the reviewer for this thoughtful assessment and constructive suggestions regarding the methodological significance of our work. We agree that entropy-regularized reweighting, covariance-based gradients, and temperature limits are established properties of exponential-family attention. We want to highlight that our contribution is a controlled investigation of how much representation adaptation is necessary when pretrained instance features are frozen.

SQA-MIL isolates this question by restricting the entire representation adapter to a single learnable query, without trainable key/value projections or additional aggregation networks. The theoretical analysis characterizes this restriction: although a query can reach any interior convex-hull mean for an individual bag, the same query must produce suitable readouts across all bags and classes, imposing an explicit expressive bottleneck.

Importantly, our experiments establish that this constraint is meaningful beyond parameter reduction. On EBRAINS-fine, SQA-MIL achieves 0.721 balanced accuracy, outperforming mean pooling (0.623), a parameter-matched sigmoid gate (0.688), and a rank-one nonlinear adapter (0.697). It also approaches DGR-MIL (0.727) using only 1,024 versus approximately 4.73 million representation parameters. On external TCGA-to-CPTAC evaluation, SQA-MIL achieves 0.931 versus 0.914 for DGR-MIL.

We further examine the limitations of this restriction through the query-count ablation (Table B.6). Increasing the number of queries from 1 to 2, 4, and 8 yields balanced accuracies of 0.718, 0.724, 0.726, and 0.724, respectively, demonstrating diminishing returns from additional adaptation capacity. Original-space representation analysis (Table B.9) further shows improved linear-probe accuracy (0.620 to 0.716), confirming that query-based reweighting produces more discriminative bag representations without modifying the pretrained encoder.

We have strengthened the main manuscript by highlighting these capacity–performance comparisons and explicitly discussing the limitations of class-shared instance selection after section 5.6. Together, these findings establish the methodological significance of SQA-MIL as a controlled and empirically testable adaptation regime, rather than a novel attention mechanism.

The statistical evidence and evaluation of the single-query bottleneck are not sufficiently complete. Although the paper reports encouraging performance, several comparisons involve relatively small differences. For example, on full-data EBRAINS-fine, SQA-MIL achieves 0.721 balanced accuracy compared with 0.727 for the strongest rich MIL baseline, while its external CPTAC improvement over DGR-MIL is 0.931 versus 0.914. Appendix B.6 describes five-seed variability and paired bootstrap confidence intervals, but the displayed result tables provide only point estimates, making it difficult to assess the reliability of these differences. Furthermore, although a class-specific-query control is introduced to examine the limitations of using a shared attention distribution across classes, its quantitative results are not reported. Since the central hypothesis concerns whether one query is sufficient for downstream adaptation, more complete statistical reporting and a direct evaluation of this expressive bottleneck are necessary to substantiate the main conclusions.

How statistically robust are the reported improvements over stronger MIL baselines? The paper describes evaluations over five matched training seeds and paired bootstrap comparisons, but the main tables primarily report point estimates without corresponding standard deviations or confidence intervals. Could the authors provide seed-level variability and paired 95% confidence intervals for the principal comparisons, particularly SQA-MIL versus DGR-MIL on EBRAINS-fine and external CPTAC? Since several performance differences are relatively small, this information would materially affect my assessment of the empirical evidence.

Our evaluation uses five matched training seeds 0–40–4, identical frozen features, and the same held-out samples for each method. We distinguish two sources of variability: variation across training seeds and uncertainty associated with the evaluation cohort. On full-data EBRAINS-fine with UNI features, SQA-MIL achieves a balanced accuracy of 0.721, compared with 0.727 for DGR-MIL. As already reported in Section 5.6, the paired difference (SQA-MIL minus DGR-MIL) is −0.006, with a 95% bootstrap confidence interval of [−0.0169, 0.0049]. This interval includes zero. We therefore do not claim a statistically established improvement or formal equivalence over DGR-MIL on this task. Instead, the relevant finding is the performance–capacity trade-off: SQA-MIL achieves competitive predictive performance using only 1,024 trainable representation parameters, compared with approximately 4.73 million for DGR-MIL, a reduction of approximately 4,620-fold.

Appendix B.6 already specifies the statistical protocol, including five matched training seeds and 10,000 paired bootstrap resamples. We address this concern by adding five-seed mean and standard deviation to the principal result tables, together with paired 95% confidence intervals for the key method comparisons in the appendix. Individual seed-level measurements are also provided in the supplementary material.

How restrictive is the class-shared single-query representation in fine-grained classification? The proposed formulation assigns the same attention distribution to all classes, which may limit its ability to identify distinct morphological evidence for different diagnoses. Although the paper introduces a class-specific-query control, its quantitative results are not clearly reported. Could the authors provide this comparison, particularly on the 30-class EBRAINS-fine task, and analyze whether different classes benefit from independent attention distributions? Such evidence would help establish whether the observed performance reflects the sufficiency of a single shared query or whether additional task-specific selection capacity is necessary.

The class-shared attention distribution is an intentional capacity constraint of SQA-MIL. Importantly, class discrimination remains class-specific through independent classifier weights.

To directly assess this limitation, we evaluate the class-specific-query control defined in Appendix B.4 on the 30-class EBRAINS-fine task. The class-specific variant achieves 0.722 balanced accuracy, compared with 0.721 for SQA-MIL, yielding a paired difference of +0.001 (95% CI: [-0.07, 0.06]), while requiring 30×30\times more representation-adaptation parameters (30,720 versus 1,024).

The nearly identical point estimates indicate no observed aggregate accuracy advantage from class-specific selection, although the confidence interval does not establish equivalence or exclude class-dependent benefits. This comparison highlights the expressivity–efficiency trade-off of single-query adaptation rather than claiming universal sufficiency.

We have added the quantitative comparison to Section 5.6 and clarified the corresponding expressive limitation in the manuscript.

Comments on the paper

1 comment
Area_Chair_XEQdOpenReview ↗

Remind to engage with author's rebuttal

Dear Reviewers,

We are now at the halfway point of the discussion phase, and the authors' rebuttal is available for review.

At this stage, I would like to request that you re-engage with the paper and the discussion thread. Please take the time to read the authors' responses, determine whether the concerns and questions you raised have been resolved, and share any further feedback with the authors as you see fit.

Your participation during the remainder of the discussion period is important for arriving at a sound final decision. Thank you for your continued contributions.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.