acceptodds
Under review as a conference paper at ICLR 2027

Cross-Modal Relational Distillation for White-Light Endoscopic Gastrointestinal Lesion Classification

Abstract

White-light imaging (WLI) is widely used for colorectal polyp assessment, whereas narrow-band imaging (NBI) reveals vascular and mucosal details but requires dedicated imaging equipment. Existing cross-modal distillation methods use verified WLI–NBI pairs for direct transfer or unpaired images for group-level supervision, leaving the relational information accessible through verified pairs underexplored. We propose Paired-anchor Cross-modal Transfer (PaCT), which uses verified pairs as shared identity anchors to transfer NBI relations to unpaired WLI images. PaCT builds an NBI teacher graph from paired and additional unpaired NBI images. For each unpaired WLI query, its affinities to WLI anchors are mapped through the verified identities to obtain a query-specific relational target from the NBI graph, without assuming a match to any unpaired NBI image. Under a common three-seed, five-fold evaluation protocol, PaCT improves mean accuracy, F1-score, and AUC over the best comparator for each metric by 2.48, 2.64, and 2.71 percentage points on PICCOLO, and by 1.93, 2.13, and 2.82 percentage points on PolypSet, respectively. PaCT thus extends the utility of verified pairs beyond paired examples while requiring only WLI at inference. An anonymous code repository is available at https://anonymous.4open.science/r/PaCT-iclr2027-code/

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Colorectal Polyp Classification, Paired-anchor Cross-modal Transfer, Unpaired-data Relational Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6hhiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how NBI images available during training can improve WLI-only lesion classification in partially paired collections. PaCT uses verified WLI–NBI pairs as shared identity anchors. A frozen NBI teacher constructs a graph from paired and additional unpaired NBI images, producing relations among the paired anchors. A frozen WLI reference model computes each unpaired WLI query’s affinities to the WLI anchors and uses the verified identities to read the NBI relations, generating a fixed query-specific distillation target. The WLI student learns from class supervision and relational distillation, while inference requires only WLI.

The paper provides a well-defined mechanism for extending the supervisory role of verified pairs to unpaired WLI images. The main unresolved issues concern the reliability of the fixed targets, the additional contribution of NBI-derived relations relative to graph-based regularization, and the correspondence between the reported experiments and the public implementation.

Critical strengths

S1. Using NBI during training to improve a classifier that requires only routine WLI at inference has clear application value.

S2. Section 3.4 connects the two modalities through verified anchor identities. This produces a well-defined target distribution over shared identities and allows unpaired WLI queries to access NBI relations without requiring direct alignment or isomorphism between the two feature spaces.

S3. Table 2 reports weighted F1 improvements of 2.64 and 2.13 percentage points over the best-performing baseline on PICCOLO and PolypSet, respectively. Table 3 examines the contribution of additional NBI graph nodes while retaining the same teacher, and Table 4 tests within-class reassignment of pair identities. These controls provide useful evidence for the roles of additional NBI observations and verified correspondence.

Critical weaknesses

W1. The additional benefit of NBI-derived relational information is not fully isolated. The paper acknowledges existing group-level transfer methods such as PaGKD, and the main comparisons support average performance gains. However, M1 replaces the relational kernel with the identity matrix, removing both NBI relations and non-trivial graph propagation. This control does not distinguish the contribution of NBI-specific relations from the benefits of graph-based regularization. A matched WLI-only graph using the same anchors, graph operators, initialization, and training budget would more directly establish the value of the cross-modal information.

W2. Target reliability depends on the frozen WLI reference affinities and the coverage of the paired anchors. Verified pairing establishes correspondence between anchor identities, but it does not ensure that a WLI query’s neighborhood identifies useful NBI relations. If these affinities are misleading or relevant anchors are absent, the resulting target can remain inaccurate throughout student training because it is cached and fixed. Table 4 supports the usefulness of correct identities, but the paper does not directly examine target quality or transfer benefits under unreliable query neighborhoods or limited anchor coverage.

W3. The statistical and data-protocol reporting limits interpretation of the comparisons. Table 2 reports standard deviations across three seed-level means, each averaged over five folds; these do not directly quantify uncertainty in paired performance differences at the lesion or patient/video-group level. Tables 3 and 4 also provide no uncertainty intervals for the ablation differences. Figure 3 shows non-monotonic point estimates on PICCOLO, whose cause cannot be determined from the reported results. In addition, Appendix A.2 describes grouping principles but does not report the per-fold counts promised in Section 4.1. More explicit pair-verification procedures and anonymized group/split manifests would support independent checking of the evaluation protocol.

W4. The public implementation does not directly support some reported experimental settings. The inspected PPR implementation rejects a restart probability of ρ = 1, although this value is used for the self-distillation ablation. The public interface also couples graph-neighborhood and bandwidth-neighborhood settings, whereas the manuscript describes a sweep that changes the graph neighborhood while retaining the bandwidth setting. The authors should identify the implementation and configurations that generated these results. These discrepancies leave the reproducibility and interpretation of the affected ablations unresolved; their implications for the main results require clarification.

Questions for authors

Q1. How much of the improvement remains against a matched WLI-only graph control using the same anchors, graph operators, initialization, and training budget?

Q2. How reliable are the transferred targets when WLI query neighborhoods are misleading or paired-anchor coverage is limited? A focused analysis would help establish the conditions for useful transfer.

Q3. Which implementation and configurations generated the ρ = 1 ablation and the graph-neighborhood sweep with fixed bandwidth settings? Please clarify their correspondence to the public release.

8 comments

W1. However, M1 replaces the relational kernel with the identity matrix, removing both NBI relations and non-trivial graph propagation. This control does not distinguish the contribution of NBI-specific relations from the benefits of graph-based regularization. A matched WLI-only graph using the same anchors, graph operators, initialization, and training budget would more directly establish the value of the cross-modal information.

Q1. How much of the improvement remains against a matched WLI-only graph control using the same anchors, graph operators, initialization, and training budget?

Response:

We thank the reviewer for suggesting the matched WLI-only graph control. We address this concern as follows. We added the requested WLI-only graph control. Both graph variants use kNN construction followed by PPR. The WLI control builds its kernel from the paired WLI anchors and forms the target as PjW=QjUW,0KWP_j^W = Q_j^{UW,0}K^W.

The WLI graph and M2 use the same verified paired-anchor identities, WLI student initialization, optimization budget, fold split, random seeds, and checkpoint-selection rule. Both graphs contain paired anchors only. Results use the same three random seeds and five-fold evaluation as the paper's ablation experiments. M1 and M2 correspond to the variants reported in Table 3 of the paper, while the WLI graph is the newly added control. M1 serves as the identity-kernel reference.

Table: WLI self-graph control on PICCOLO and PolypSet.

DatasetSettingAcc (%)F1 (%)AUC
PICCOLOM172.4669.8573.42
PICCOLOWLI graph73.4871.1774.53
PICCOLOM277.3374.9976.66
PolypSetM183.9683.7586.04
PolypSetWLI graph85.5885.4086.33
PolypSetM287.3687.3488.34

The WLI graph improves over M1 on both datasets. Relative to the WLI graph, M2 improves ACC, weighted F1, and AUC by 3.85, 3.82, and 2.13 percentage points on PICCOLO, and by 1.78, 1.94, and 2.01 percentage points on PolypSet. Both graphs contain paired anchors only. The results support a benefit from NBI-derived relations beyond the tested WLI graph propagation.

W2. Table 4 supports the usefulness of correct identities, but the paper does not directly examine target quality or transfer benefits under unreliable query neighborhoods or limited anchor coverage.

Q2. How reliable are the transferred targets when WLI query neighborhoods are misleading or paired-anchor coverage is limited? A focused analysis would help establish the conditions for useful transfer.

Response:

We thank the reviewer for highlighting the importance of target reliability and anchor coverage. We address these concerns as follows.

1. Reliability on misleading WLI queries.

We analyzed the training-time targets offline under seed 2026. We counted each unpaired WLI image once per fold if it entered at least one residual distillation batch. This included 3,793 of 3,859 candidate image–fold records on PICCOLO and all 1,573 on PolypSet. The 66 omitted PICCOLO records had valid cached targets and zero exposures. All three retention levels use the same query set. We average the diagnostic metrics equally over five folds.

For each query, we sum the anchor weights in Q and P by class and compare the highest-weight class to the ground truth. In Table 1, Q error is the initial query error rate; Δ True is the change in true-class weight among initially incorrect queries. The correction rate is measured among initially incorrect queries, and the new error rate is measured among initially correct queries. P wrong >50% is the fraction of all queries for which one incorrect class receives strictly more than 50% of P.

Table 1: Target reliability on WLI queries with 100% anchors.

DatasetQ error (%)Δ True (pp)Correction rate (%)New error rate (%)P wrong >50% (%)
PICCOLO6.01+1.6571.610.305.94
PolypSet17.38+1.2760.170.1017.32

Propagation increases the average true-class weight for initially misranked queries, although most retain an incorrect top-ranked class. On PolypSet, one 50/50 tie was counted as incorrect by argmax and fell outside the strict majority criterion.

2. Target reliability under reduced anchor retention.

We used nested, class-stratified banks at 25%, 50%, and 100% nominal retention. At reduced retention levels, coverage of every lesion or sequence-proxy group represented in the full bank was not guaranteed. Within each fold and seed, the frozen NBI teacher, WLI reference, and student initialization were shared. The NBI graph used the selected paired NBI anchors and the original unpaired NBI pool. All original paired WLI images remained on the paired CE route, and the original unpaired WLI queries remained on the CE + KL route. The graph and targets were recomputed for each bank.

Table 2 reports training-target diagnostics under seed 2026, averaged over five folds. Pairs is the mean number of retained anchor pairs; Δ True is computed on each setting's own initially misranked queries.

Table 2: Target diagnostics under different anchor retention rates on PICCOLO and PolypSet.

DatasetRetentionPairsQ error (%)Δ True (pp)
PICCOLO25%183.66.71+0.906
PICCOLO50%366.86.25+1.012
PICCOLO100%732.86.01+1.657
PolypSet25%30.018.28+0.116
PolypSet50%59.817.83+0.432
PolypSet100%119.017.38+1.276

Table 3 reports held-out AUC at different anchor retention rates from the separate student ablation, averaged over three seeds and five folds, using the full test WLI set in each fold.

Table 3: Held-out AUC on PICCOLO and PolypSet.

Dataset25%50%100%
PICCOLO75.6777.1678.23
PolypSet88.9890.0992.03

As shown in Table 2, with fewer anchors, initial errors were more frequent and the average true-class weight gain on each setting's initially misranked queries was smaller. Table 3 shows that held-out AUC was higher with more retained anchors on both datasets. These results characterize the effect of reducing within-group anchors while preserving group coverage.

W3. The statistical and data-protocol reporting limits interpretation of the comparisons. Table 2 reports standard deviations across three seed-level means, each averaged over five folds; these do not directly quantify uncertainty in paired performance differences at the lesion or patient/video-group level. Tables 3 and 4 also provide no uncertainty intervals for the ablation differences. Figure 3 shows non-monotonic point estimates on PICCOLO, whose cause cannot be determined from the reported results. In addition, Appendix A.2 describes grouping principles but does not report the per-fold counts promised in Section 4.1. More explicit pair-verification procedures and anonymized group/split manifests would support independent checking of the evaluation protocol.

Response:

We thank the reviewer for the questions about statistical uncertainty and the data protocol. We clarify these points as follows.

1. Interpretation of statistical results.

The standard deviations in Table 2 summarize variability across three training seeds, after averaging each seed's results over five folds. They do not quantify uncertainty in paired between-method differences over held-out groups. The differences in Tables 3 and 4 are reported as point estimates.

2. Clarification of the results in Figure 3.

The sweep varies the fraction of unpaired NBI images included as graph nodes, while teacher pretraining, the paired anchors, and the WLI training pool remain fixed. On PICCOLO, using 25% and 100% of the unpaired NBI pool as graph nodes yields similar accuracy (78.05% vs. 78.13%), while AUC increases from 77.53% to 78.23%. The reported sweep does not establish the cause of the non-monotonic intermediate results.

3. Per-fold counts and data protocol.

Table 1 below provides the per-fold counts for Appendix A.2. Parentheses give class counts in the order Hyperplasia / Adenoma / Adenocarcinoma for PICCOLO and Hyperplasia / Adenoma for PolypSet. Training pairs are counted as pairs. The validation and test columns include all WLI images assigned to those sets.

Table 1 Per-fold data counts on PICCOLO and PolypSet.

DatasetFoldTrain pairsTrain unpaired WLITrain unpaired NBIVal WLITest WLI
PICCOLO0737 (189/422/126)754 (118/610/26)178 (34/123/21)193 (26/144/23)436 (71/278/87)
PICCOLO1765 (188/468/109)721 (107/515/99)147 (18/108/21)193 (34/136/23)441 (75/335/31)
PICCOLO2716 (183/450/83)792 (98/595/99)187 (29/153/5)196 (34/131/31)416 (89/278/49)
PICCOLO3727 (195/438/94)799 (100/602/97)177 (24/137/16)189 (34/132/23)405 (75/282/48)
PICCOLO4719 (157/454/108)793 (125/584/84)190 (34/135/21)186 (28/135/23)422 (94/281/47)
PolypSet0120 (48/72)311 (135/176)350 (138/212)58 (26/32)126 (54/72)
PolypSet1120 (48/72)307 (133/174)343 (139/204)60 (25/35)128 (57/71)
PolypSet2117 (45/72)322 (141/181)343 (159/184)54 (22/32)122 (55/67)
PolypSet3123 (51/72)319 (143/176)313 (149/164)58 (26/32)115 (43/72)
PolypSet4115 (45/70)314 (135/179)337 (158/179)62 (29/33)124 (54/70)

The updated anonymous release includes the actual fold manifests and training-pair ledgers under splits/grouped_v1/. The manifests record split membership, group aliases, modality, label, and pair ID. The ledgers identify the WLI and NBI endpoints of every training pair.

Reviewer_GdX7OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses polyp classification from white-light imaging (WLI) when narrow-band imaging (NBI) is available only during training and only part of the training set has verified WLI–NBI pairs. This is plausible to clinical settings in real time operations. Method is called PaCT, and it builds a frozen NBI graph over paired and unpaired NBI images. It runs a kind of PageRank from each paired NBI anchor and restricts the endpoints to paired anchors. Then a transformer is built upon. There is also teacher-student network included. Experiments use five-fold, video-grouped cross-validation with three seeds on PICCOLO (three classes) and a binary "PolypSet" protocol, against eight reimplemented baselines. Some promising results are obtained.

Critical strengths
  • The anchor-reading construction is simple, cheap, and clearly specified, and the staged ablation is well designed.
  • Real time classification can be very useful in clinic.
Critical weaknesses
  • The paper's central claim is that NBI relations, routed through verified identities, drive the gains. The evidence does not prove that. For example, even randomly reassigning identities within class retains 66% of the F1 gain on PICCOLO, and that randomized variant alone beats every baseline. This is showing that the problem maybe easier than it looks and no need to combine several engineeringly designed methods to solve. You do not cut fish with sword , but with knife (as an anecdote).
  • No control replaces the NBI graph with a WLI or label-derived graph.
  • The comparisons also rest on hyperparameter defaults that coincide with the test-F1 maximum in all four sweeps, a distillation weight that differs 117-fold between datasets without a reported search, and a baseline protocol that the appendix contradicts.
  • Two datasets with roughly 40-60 patients are not generalizable. Rigor is weak, experiments are not convincing, method's generalizability is questionable.
  • The source of the gain is unidentified. No control uses a WLI-anchor graph, a label-block graph, or a random-feature kNN graph.
  • The PolypSet construction carries a leakage risk. this may inflate the results.
  • novelty is combination of all available methods presented in the manuscript. Those methods are already in the literature. Combined engineering is marginally innovative.
Questions for authors

Updated review: Some of the concerns were addressed, and I have increased my score accordingly but majority of the questions remained unclear, experiments do not support the claims fully.

Previous review: The randomized-identity variant reaches 73.80 F1 on PICCOLO, above every baseline. What fraction of PaCT's advantage over prior work do the authors attribute to NBI-specific relations rather than to anchor-space distillation?

The introduction emphasizes extending supervision through unpaired NBI. On PICCOLO, unpaired nodes add 0.88 F1 while the paired-only graph adds 5.14. Which is the paper's primary contribution?

How does the setting relate to SMIL and other partial-modality formulations?

What is the descriptor dimension d? What is the typical entropy, or effective support size, of Q^{UW,0} on each dataset?

What is the distribution of K^N diagonal mass at ρ = 0.15, and how often does a row place majority mass on anchors of a different class?

For what fraction of unpaired WLI images does P_j place majority class mass on the wrong class? How does performance on those images change after training?

How were WLI–NBI pairs "verified" on each dataset?

Are the PolypSet pairs the PolypsAlign ISIT-UMR collation?

From which sources do the unpaired images come?

How were lesions deduplicated across ISIT-UMR/GLRC and the other PolypSet sources?

How many lesions and patients per class appear in each test fold?

How is macro one-vs-rest AUC computed in folds missing a class?

Where are the fold counts promised for Appendix A.2?

Why do four NBI-using baselines fall below the CE-continuation control M0 on PICCOLO accuracy?

Which attribution method produces Figure 4, and how were the four cases chosen?

37 comments

Q1. The randomized-identity variant reaches 73.80 F1 on PICCOLO, above every baseline. What fraction of PaCT's advantage over prior work do the authors attribute to NBI-specific relations rather than to anchor-space distillation?

Response:

We thank the reviewer for this question. The randomized-identity variant retains the NBI relation kernel while shuffling the WLI–NBI anchor correspondences within each class. To quantify the additional benefit of using NBI relations, we added a matched WLI self-graph control between M1 and M2.

All three variants distill distributions over the same paired anchors. M1 uses the initial WLI anchor affinities as targets, while the WLI self-graph and M2 propagate these affinities through graphs constructed from the paired WLI and NBI anchors, respectively.

Table 1. Weighted F1 of graph controls on PICCOLO.

VariantWeighted F1 (%)Change (pp)
M169.85—
WLI self-graph71.17+1.32
M274.99+3.82

As shown in Table 1, adding WLI graph propagation improves F1 by 1.32 percentage points over M1. With the paired anchors and graph settings fixed, switching the graph's relation source from WLI to NBI further improves F1 by 3.82 points. This comparison supports an additional benefit from NBI-derived relations beyond the tested WLI graph propagation.

The advantage over prior methods compares complete systems and cannot be uniquely partitioned into a fraction attributable to NBI relations and a fraction attributable to anchor-space distillation. With anchor-space distillation held fixed, the matched graph control estimates a 3.82-percentage-point gain in weighted F1 from using paired NBI relations in place of paired WLI relations.

Q2. The introduction emphasizes extending supervision through unpaired NBI. On PICCOLO, unpaired nodes add 0.88 F1 while the paired-only graph adds 5.14. Which is the paper's primary contribution?

Response:

We thank the reviewer for asking us to clarify the primary contribution. PaCT's primary contribution is a framework that uses verified paired anchors to transfer NBI relational information to unpaired WLI queries. In M2, the NBI graph contains paired anchors only, while the NBI teacher is pretrained using both paired and unpaired NBI images. M3 extends this graph by adding unpaired NBI images as nodes.

Table 1. Unpaired NBI availability and weighted F1 on PICCOLO and PolypSet.

DatasetUnpaired NBI (%)M1M2M3M2 − M1M3 − M2
PICCOLO16.1–20.969.8574.9975.87+5.14+0.88
PolypSet71.8–74.683.7587.3490.87+3.59+3.53

All performance values are weighted F1 percentages, averaged over three seeds and five folds; differences are in percentage points. Unpaired NBI (%) is the fraction of unpaired images among all training NBI images, reported as the range across five folds. For each fold, the denominator includes one NBI image per training pair and all unpaired training NBI images.

As shown in Table 1, M2 improves over M1 on both datasets. M3 further improves weighted F1 by 0.88 points on PICCOLO and 3.53 points on PolypSet. Since unpaired NBI images are already used in teacher pretraining for M2, the M3-to-M2 difference measures the additional benefit of including them as graph nodes, rather than their total contribution to the training pipeline. The larger gain on PolypSet is observed alongside a larger proportion of unpaired NBI images in its training pool. This provides context for the difference, although the datasets also differ in other respects.

The M1-to-M2 comparison combines the effects of graph propagation and NBI relations. As discussed in our response to Q1, the matched WLI self-graph control further assesses the contribution of NBI-derived relations.

Q3. How does the setting relate to SMIL and other partial-modality formulations?

Response:

We thank the reviewer for asking us to clarify the relationship between PaCT and partial-modality learning. We position our setting and method in relation to these formulations as follows.

Relation to missing-modality learning. SMIL addresses missing modalities during training, testing, or both using Bayesian meta-learning. PaCT considers a specific asymmetric setting: the training pool contains verified WLI–NBI pairs as well as unpaired images from both modalities, while the classifier uses WLI alone at inference. Ma et al., SMIL, AAAI 2021

Auxiliary modality during training. RD-Net uses depth information to guide training and requires only RGB images at inference for polyp segmentation. PaCT shares this separation between training and inference inputs, but uses NBI as the auxiliary modality for WLI-based lesion classification. Li et al., RD-Net, CVPR 2025

Role of paired anchors. UML uses parameter sharing to learn from unpaired auxiliary modalities without requiring sample-level pairs. PaCT instead uses the available verified pairs to establish corresponding WLI and NBI anchors. Each unpaired WLI query is represented by its affinities to the WLI anchors; the corresponding NBI anchors then provide an index into the NBI relation kernel. This allows NBI relational information to supervise WLI queries without directly paired NBI images. Gupta et al., UML, ICLR 2026

Reviewer_YD7iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the classification problem involving partially paired WLI–NBI data. In addition to verified WLI–NBI pairs, the dataset includes observations consisting of either WLI or NBI data alone. The PaCT method treats each verified pair as an anchor for a shared identity. During inference, the system employs a fixed WLI reference model to compute the affinity QQ between an unpaired WLI query sample and these anchors. Subsequently, leveraging the verified identity mappings, this is transformed into an association matrix KNK^N—derived via the Personalized PageRank algorithm applied to a graph constructed from both paired and unpaired NBI data. This yields an association target PP for a given query, which is then distilled into a WLI student model using Kullback–Leibler (KL) divergence. At inference time, only the WLI modality is utilized.

Critical strengths

This paper proposes a solution to a practical issue that has been overlooked in previous research. Clinical datasets often exhibit a partially paired structure; existing methods either rely exclusively on paired data (e.g., PolypsAlign, CPC-Trans, ADD) or disregard the identity correspondences between samples entirely (e.g., PaGKD, Better Together). PaCT effectively fills this research gap. The design avoids the artificial construction of cross-modal correspondences. Confirmed identity correspondences are used solely as indices, while unpaired NBI data serve only to expand the topology on the teacher side—ensuring that no "pseudo-pairings" are created between WLI and NBI modalities. The rationale behind this design is clear. Furthermore, the paper explicitly addresses the limiting case where ρ→1\rho \to 1 (which reduces to WLI self-distillation, as shown in Equation 6). The experimental protocol is notably rigorous. Information leakage is prevented through a grouping strategy based on VP (lesion/site).

Critical weaknesses
  1. The gains cannot be attributed to NBI modality knowledge versus graph-smoothing regularization. The target P_j = Q_j^{UW,0} · M K^N Mᵀ is essentially the product of the WLI anchor affinities Q and the NBI relation matrix K^N. If the WLI-space relation matrix K^W (constructed from the same paired-WLI descriptors with the identical kNN + PPR procedure) is highly similar to K^N, then the whole "cross-modal transfer" degenerates into "graph smoothing of WLI affinities," with NBI acting merely as a regularizer. The 4.87-point jump from M1 (ρ=1, pure anchor self-distillation, no NBI) to M2 (NBI graph added), 72.46 → 77.33 in Table 3, cannot distinguish these two explanations at all. Please report: (a) the agreement between K^W and K^N (Spearman correlation, rank overlap, Jaccard of neighbor sets for the same lesion across modalities); (b) a control that replaces the NBI graph with the WLI self-graph​ (P = Q K^W). If that control lands close to PaCT, the paper's central claim — transferring NBI relational evidence to WLI — does not hold, and novelty is substantially weakened.
  2. In addition, the contribution of unpaired NBI on PICCOLO is small: M2 77.33 → M3 78.13 is only +0.80, and Figure 3 shows saturation at just 25% of the unpaired NBI pool (78.05 / 77.53 versus 78.13 / 78.23 for the full pool). The gains are concentrated on the much smaller PolypSet (165 pairs / 450 unpaired NBI).
  3. Sensitivity to the anchor count K is unknown, yet this is precisely the paper's motivation; external validation is missing.
Questions for authors

See Critical Weaknesses

4 comments

W1. The gains cannot be attributed to NBI modality knowledge versus graph-smoothing regularization. The target P_j = Q_j^{UW,0} · M K^N Mᵀ is essentially the product of the WLI anchor affinities Q and the NBI relation matrix K^N. If the WLI-space relation matrix K^W (constructed from the same paired-WLI descriptors with the identical kNN + PPR procedure) is highly similar to K^N, then the whole "cross-modal transfer" degenerates into "graph smoothing of WLI affinities," with NBI acting merely as a regularizer. The 4.87-point jump from M1 (ρ=1, pure anchor self-distillation, no NBI) to M2 (NBI graph added), 72.46 → 77.33 in Table 3, cannot distinguish these two explanations at all. Please report: (a) the agreement between K^W and K^N (Spearman correlation, rank overlap, Jaccard of neighbor sets for the same lesion across modalities); (b) a control that replaces the NBI graph with the WLI self-graph (P = Q K^W). If that control lands close to PaCT, the paper's central claim — transferring NBI relational evidence to WLI — does not hold, and novelty is substantially weakened.

Response:

We thank the reviewer for the detailed comments and for suggesting the WLI graph control. We address this concern as follows.

(a) Agreement between WLI and NBI graphs.

We aligned anchors by verified pair ID and excluded self-relations. Rowwise Spearman and top-kk overlap use common paired candidates. Overlap is the shared fraction of the top-kk sets, averaged over uniform tie-breaking at the cutoff. We average over anchors within each fold and then across five folds. Jaccard uses directed pre-PPR 5NN sets for the paired graphs.Table 1 compares the paired-WLI graph with the NBI graphs used in M2 and M3.

Table1: Agreement between WLI and NBI graphs on PICCOLO and PolypSet.

DatasetNBI graphSpearmanOverlap@5@10@205NN Jaccard
PICCOLOM20.6080.26170.32460.38080.171
PICCOLOM30.6010.26350.32880.3837—
PolypSetM20.4670.22830.26870.35750.140
PolypSetM30.3940.25240.27960.3444—

The paired graphs share rank structure, with 26.17% and 22.83% top-five overlap on PICCOLO and PolypSet (uniform-selection references of 0.68% and 4.24%). M3 adds unpaired NBI to the pre-PPR candidate set.

(b) WLI self-graph control.

The control uses a kNN + PPR graph over paired WLI anchors, with Pj=QjUW,0KWP_j=Q_j^{UW,0}K^W. It shares the anchor identities, initialization, training budget, and selection rule with M2. Table 2 reports results over five folds using the same three random seeds as in the paper's ablation experiments.

Table2: WLI self-graph control on PICCOLO and PolypSet.

DatasetSettingACC (%)F1 (%)AUC
PICCOLOM172.4669.8573.42
PICCOLOWLI graph73.4871.1774.53
PICCOLOM277.3374.9976.66
PolypSetM183.9683.7586.04
PolypSetWLI graph85.5885.4086.33
PolypSetM287.3687.3488.34

M2 improves accuracy over the WLI graph by 3.85 points on PICCOLO and 1.78 on PolypSet, supporting a benefit from NBI relations beyond WLI graph propagation.

W2. In addition, the contribution of unpaired NBI on PICCOLO is small: M2 77.33 → M3 78.13 is only +0.80, and Figure 3 shows saturation at just 25% of the unpaired NBI pool (78.05 / 77.53 versus 78.13 / 78.23 for the full pool). The gains are concentrated on the much smaller PolypSet (165 pairs / 450 unpaired NBI).

Response:

We thank the reviewer for raising this question about the contribution of unpaired NBI on the two datasets. We clarify this point as follows. PICCOLO has 147–190 unpaired NBI images per training fold (0.19–0.26 per anchor), compared with 313–350 on PolypSet (2.54–2.93 per anchor). This difference may help contextualize the larger M2-to-M3 gains on PolypSet, although it does not by itself establish the cause of the difference.

M3 vs M2

DatasetACC(%)F1(%)AUC
PICCOLO+0.80+0.88+1.57
PolypSet+3.26+3.53+3.69

Unpaired NBI nodes: 25% → 100%

DatasetACC (%)AUC
PICCOLO78.05 → 78.1377.53 → 78.23
PolypSet88.97 → 90.6289.82 → 92.03

We agree that the M2-to-M3 accuracy gain is smaller on PICCOLO, and that 25% of its unpaired NBI nodes already achieves accuracy close to the full-pool result. The full pool yields a 0.70-point higher AUC. On PolypSet, increasing the node fraction from 25% to 100% yields larger gains in both accuracy and AUC.

The M2–M3 differences are point estimates obtained using the same three random seeds, each evaluated over five folds. In Figure 3, only the unpaired NBI nodes included in the graph are varied; teacher pretraining, paired anchors, and WLI data remain fixed.

W3. Sensitivity to the anchor count K is unknown, yet this is precisely the paper's motivation; external validation is missing.

Response:

We thank the reviewer for highlighting the importance of anchor-count sensitivity and external validation. We address these concerns as follows.

1.Sensitivity to the anchor count K

We used nested, class-stratified subsets of the training anchor pairs at nominal retention levels of 25%, 50%, and 100%. The NBI teacher was pretrained on the full training-fold NBI pool, including the NBI images of pairs later omitted from the anchor bank. Within each fold and seed, all retention settings shared the same frozen NBI teacher, frozen WLI reference, and WLI student initialization.

WLI images from all original pairs, including omitted pairs, stayed on the paired CE-only route. The original unpaired WLI queries stayed on the CE + KL route. Each NBI graph contained the selected paired NBI anchors and the original unpaired NBI pool. The NBI images of omitted pairs were excluded from the graph. We recomputed anchor statistics, bandwidths, query affinities, the NBI graph and PPR kernel, and the distillation targets for each anchor bank.

Table 1: Anchor-count sensitivity on PICCOLO and PolypSet.

Datasetretentionpairsimages (%)Acc (%)F1 (%)AUC
PICCOLO25%183.615.2274.7372.2275.67
PICCOLO50%366.830.4077.4474.2977.16
PICCOLO100%732.860.7478.1375.8778.23
PolypSet25%30.06.7486.3186.1688.98
PolypSet50%59.813.4487.0287.1690.09
PolypSet100%119.026.7590.6290.8792.03

Retention is the nominal percentage of the full paired-anchor bank retained in each fold, with the corresponding pair count KfK_f rounded up. Pairs is the mean number of retained anchor pairs across five folds. Images (%) is computed as 100×2Kf/Nf100\times 2K_f/N_f within each fold and then averaged over five folds. Each retained pair contributes two images, and NfN_f is the fixed number of all WLI and NBI images in that fold's original training pool. Performance uses three seeds and five folds.

As shown in Table 1, on PICCOLO, half the anchors yield an AUC within 1.07 points of the full model. On PolypSet, the 25% setting retained an average of 30 anchor pairs per fold and achieved a held-out AUC of 88.98%.

2.External validation

For external validation, we used ERCPMP (Mendeley Data V8) exclusively for final evaluation. Training, hyperparameter tuning, and checkpoint selection used PolypSet only. We manually verified that all 233 external images were WLI. The subset represents 72 unique patient codes, with 52 hyperplastic polyp images (14 codes) and 181 adenoma images (58 codes). Adenomas include tubular, tubulovillous, and villous subtypes.

For each variant, we directly evaluated the same 15 PolypSet checkpoints used in the corresponding ablation results (five folds across three random seeds) on the same 233 external images, without any fine-tuning on ERCPMP. We computed metrics separately for each checkpoint and averaged the 15 metric values. The WLI graph denotes the paired-WLI graph control introduced in our response to W1.

Table 2: External evaluation on ERCPMP

PolypSet-trained variantAUCF1 (%)
M066.2867.19
M168.9671.05
WLI graph71.3172.90
M278.5278.38
M3 (PaCT)81.8982.25

As shown in Table 2, M2 and M3 exceed the WLI graph by 7.21 and 10.58 AUC points on this external binary task. PaCT uses the student classifier at inference. The NBI graph and query-to-anchor affinities construct the training targets.

Dear Reviewer YD7i,

Thank you again for your careful review and constructive suggestions. We have posted detailed responses to W1–W3 in the discussion. As the deadline approaches, we would like to briefly summarize the additional results.

  1. WLI–NBI relation agreement (W1). We aligned anchors by verified pair identity and reported Spearman correlation, top-k overlap, and Jaccard similarity of the original 5-nearest-neighbor sets. For the paired graphs, top-5 overlap was 26.2% on PICCOLO and 22.8% on PolypSet.

  2. Matched WLI self-graph control (W1). With the paired anchors, graph settings, and student training held fixed, replacing WLI relations with paired NBI relations improved weighted F1 by 3.82 percentage points on PICCOLO and 1.94 points on PolypSet.

  3. Contribution of unpaired NBI nodes (W2). With the pretrained teacher and paired anchors fixed, adding the full unpaired NBI pool to the graph improved accuracy by 0.80 points on PICCOLO and 3.26 points on PolypSet. On PICCOLO, the 25% and full-pool settings had similar accuracy, while the full pool yielded 0.70 points higher AUC.

  4. Sensitivity to anchor count (W3). We evaluated 25%, 50%, and 100% retention of the paired anchors and reported the share of the original training image pool represented by the retained anchors.

  5. External WLI-only validation (W3). We evaluated the PolypSet-trained variants on 233 ERCPMP WLI images without fine-tuning or external model selection. On this external binary task, PaCT improved ROC-AUC by 10.58 points and weighted F1 by 9.35 points over the WLI self-graph control.

We would appreciate hearing whether our responses address your concerns. If any point would benefit from further clarification, we would be glad to follow up during the remaining discussion period.

Thank you for your time and consideration.

Best regards,
The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal Summary

Dear ICLR 2027 AC, SAC, and PC,

Thank you for your time and for coordinating the review process. We also thank the reviewers for their careful feedback and constructive discussion. To facilitate your assessment, we summarize the core contributions of PaCT and the main evidence and clarifications added during the rebuttal below.

Core Contributions of PaCT

PaCT addresses a cross-modal learning setting in which training data contain verified WLI–NBI pairs as well as unpaired images from both modalities, while the deployed classifier receives WLI alone. Our contributions are summarized as follows:

  • Identity-indexed relational transfer. PaCT reuses verified WLI–NBI pairs as shared identity anchors. An unpaired WLI query is represented by its affinities to the WLI anchors; the verified identities connect these affinities to the corresponding NBI anchors and their relations. This gives the query an NBI-derived training target without requiring its own NBI counterpart.

  • Extended use of partially paired data. Additional unpaired NBI images can enrich the teacher graph without introducing unverified WLI–NBI matches. In this way, verified pairs extend NBI relational supervision beyond their directly paired samples, while inference uses only the trained WLI classifier.

  • Performance under WLI-only inference. Under a common three-seed, five-fold evaluation, PaCT improves weighted F1 over the strongest evaluated baseline by 2.64 percentage points on PICCOLO and 2.13 points on PolypSet. An external WLI-only evaluation further examines the trained models on ERCPMP (Section 4.3, Table 2; Section 4.5, Table 8).

  • Evidence for the transfer mechanism. Controlled comparisons examine NBI versus WLI relation sources, within-class anchor weighting, and verified identity mapping. These analyses provide complementary evidence under their respective graph settings.

Summary of Rebuttal-Driven Revisions

In direct response to the reviewers' comments, we have revised and re-uploaded the manuscript, with the revised and newly added text highlighted in blue. These revisions strengthen the experimental evaluation and clarify the data protocol, implementation, and presentation. We also conducted additional follow-up analyses during the discussion. We summarize the main changes and findings below, with pointers to the revised manuscript and detailed responses.

1. Strengthened Evidence for the Transfer Mechanism

  • WLI self-graph control. To examine NBI relations beyond WLI graph propagation, we added a WLI self-graph built on the same paired anchors. M2 improves weighted F1 over the default WLI graph by 3.82 and 1.94 points on PICCOLO and PolypSet. In a follow-up, we separately tuned the WLI graph; the respective differences remain 3.29 and 1.39 points. The sweep and validation-based configuration selection are described in the response. (Reviewers YD7i W1, 6hhi W1/Q1, and GdX7 Q1; revised manuscript, Section 4.4, Table 4; graph-control follow-up.)

  • Class-level structure and within-class relations. For full PaCT, we evaluated label-block and random-feature graph controls using the same paired and unpaired NBI nodes. In a follow-up, we added a matched label kernel for M2 that preserves each row's diagonal and per-class mass while making non-self weights uniform within each class. M2 improves weighted F1 over this matched control by 1.92 and 1.90 points on PICCOLO and PolypSet, respectively. (Reviewer GdX7 W2/W5 and Q5/Q6; revised manuscript, Section 4.4, Table 5; matched-kernel follow-up.)

  • Verified identity mapping. To examine correspondence beyond class membership, we compared verified identities with within-class reassignment on the same full graph. Verified identities improve weighted F1 over random reassignment by 2.07 and 3.67 points on PICCOLO and PolypSet. The paired-graph, matched-kernel, and full-graph identity comparisons address distinct aspects of the mechanism. (Reviewer GdX7 W1/Q1; revised manuscript, Section 4.4, Table 6.)

AuthorsOpenReview ↗

Summary of Rebuttal-Driven Revisions (continued)

2. Expanded Evaluation

  • External WLI-only evaluation. We evaluated PolypSet-trained variants on 233 ERCPMP WLI images from 72 distinct patient codes, without fine-tuning or external-set model selection. For each variant, we evaluated the 15 checkpoints from three seeds and five folds separately and averaged their image-level metrics. PaCT improves ROC-AUC and weighted F1 over the WLI self-graph by 10.58 and 9.35 percentage points, respectively. (Reviewers YD7i W3 and GdX7 W4; revised manuscript, Section 4.5, Table 8; external-validation response.)

  • Anchor availability. We evaluated 25%, 50%, and 100% retention of the paired-anchor bank, recomputing the graph and targets and retraining the student for each setting. At 25% retention, held-out AUC is 75.67% on PICCOLO and 88.98% on PolypSet. We clarified that the PolypSet setting also reduces source-sequence coverage: its retained anchors cover 29–31 of the 43–45 groups represented in the full training anchor bank per fold. (Reviewers YD7i W3 and 6hhi W2/Q2; revised manuscript, Section 4.4, Table 7 and Appendix A.7; coverage clarification.)

  • Additional unpaired NBI nodes. M2 and M3 share a teacher pretrained on paired and unpaired NBI images. With this teacher fixed, adding the unpaired NBI pool as graph nodes improves accuracy over M2 by 0.80 points on PICCOLO and 3.26 points on PolypSet. This comparison measures the additional contribution of those nodes to the graph. (Reviewers YD7i W2 and GdX7 Q2; revised manuscript, Section 4.4, Figure 3.)

3. Target Behavior and Reliability

  • Affinity and target diagnostics. We reported anchor-bank composition, affinity entropy, and same-class mass before and after propagation. On the same exposed training queries, propagation increases mean same-class mass by 3.38 points on PICCOLO and 8.94 points on PolypSet. These diagnostics use seed 2026 and complement the retrained graph controls. (Reviewers 6hhi W2/Q2 and GdX7 Q4–Q6; revised manuscript, Appendix A.10; affinity-statistics follow-up.)

  • Wrong-class-majority subsets. We compared M0 and PaCT on fixed training-query subsets and reported a post-hoc test-set breakdown for M0, M1, and PaCT using the same target-based flag. For this diagnostic, test targets are computed offline using the corresponding fold's training-derived reference, anchors, and NBI graph; test labels define the flag afterward. The analysis uses seed 2026 and does not change WLI-only inference. (Reviewer GdX7 Q6; subset analysis.)

AuthorsOpenReview ↗

Summary of Rebuttal-Driven Revisions (continued)

4. Improved Reproducibility and Clarity

  • Data provenance and grouped evaluation. We clarified the sources of paired and unpaired images, pair verification, fold composition, and recorded group boundaries. The anonymous release provides source records for all 1,230 PolypSet images, fold manifests, and training-pair lists. Recorded groups and proxies are kept within a split; the available source identifiers do not establish whether separate releases contain the same clinical lesion or patient. (Reviewers 6hhi W3 and GdX7 W6/Q7–Q13; revised manuscript, Appendix A.2; anonymous release.)

  • Baseline and evaluation protocol. We clarified the common folds, WLI training pool, validation-based checkpoint selection, and method-specific configurations. The main comparison reports statistics across three seed-level five-fold means; the graph-control follow-up additionally reports variability and paired comparisons across the 15 seed–fold results. (Reviewer GdX7 W3/Q1/Q14; revised manuscript, Sections 4.1–4.3 and Appendix A.6; graph-control follow-up.)

  • Implementation and ablation settings. We traced M1 to the original direct target construction and clarified its identity-kernel equivalence. We also documented the graph-neighborhood sweep with a fixed bandwidth neighborhood and updated the anonymous interface to expose these settings. (Reviewer 6hhi W4/Q3; revised manuscript, Appendices A.4–A.6; implementation clarification.)

During the rebuttal, we addressed the reviewers' feedback through additional controls, external evaluation, and clarifications to the data and implementation protocols. We appreciate that Reviewers 6hhi and GdX7 increased their ratings following the discussion.

PaCT provides an identity-indexed mechanism for extending NBI relational supervision to unpaired WLI queries. Together, the controlled comparisons and external WLI-only evaluation offer complementary evidence for this mechanism and its utility in the evaluated settings.

We sincerely thank the AC, SAC, PC, and all reviewers for their time and valuable suggestions. Their feedback has helped us strengthen the evaluation and presentation of PaCT.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.