acceptodds
Under review as a conference paper at ICLR 2027

TriEKB-Surg: Triadic Evolutionary Knowledge Bridging for Task-Heterogeneous Federated Surgical Video Understanding

Abstract

Federated learning (FL) enables institutions to collaboratively train surgical video understanding models without sharing raw data. However, most existing methods assume that all clients optimize the same task, which is unrealistic when institutions provide different forms of supervision. Although heterogeneous-task updates may share the same parameter space and thus be numerically aggregatable, they arise from distinct prediction spaces and optimization objectives. Therefore, parameter aggregatability does not guarantee knowledge transferability: a source update may be semantically irrelevant to the target, unreliable at the current training stage, or incompatible with the target objective. To address this challenge, we propose **Tri**adic **E**volutionary **K**nowledge **B**ridging (**TriEKB-Surg**), which frames task-heterogeneous FL as an evolving and client-specific knowledge-transfer problem. The triadic design considers semantic relevance, optimization-stage reliability, and task-compatible absorption. At the server, dual-prompt compatibility estimates cross-client relevance at coarse task-family and fine task-semantic levels, while an evolutionary curriculum combines source reliability with target-specific update agreement to construct personalized bridges. On the client side, cross-task alignment transforms transferred knowledge into task-compatible auxiliary supervision, while task-aware feature modulation mitigates optimization conflict. By determining *which* knowledge to transfer, *when* to trust it, and *how* to absorb it, TriEKB-Surg enables adaptive collaboration across heterogeneous surgical tasks. Experiments on five public surgical video datasets show that TriEKB-Surg achieves the best overall performance among the evaluated FL methods. Code will be made publicly available.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated learning, surgical video understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tMvhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates federated surgical video understanding when clients perform different prediction tasks. The authors propose TriEKB-Surg, which combines prompt-based client compatibility, reliability- and update-agreement-based aggregation, and local cross-task alignment. They evaluate the approach on three segmentation datasets and two depth-estimation datasets, reporting an aggregate relative improvement of 15.76% over FedAvg. The study contributes a framework for personalized knowledge transfer across heterogeneous surgical tasks.

Critical strengths
  1. The paper addresses differences in institutional annotation types, extending the problem formulation beyond federated learning with a shared prediction task.
  2. The framework combines semantic compatibility, source reliability, and cross-task alignment, addressing both source selection and local knowledge absorption within one federated procedure.
  3. Comparisons with federated baselines and component ablations across five datasets provide empirical support for the combined framework on segmentation and depth estimation.
Critical weaknesses
  1. The ablation study does not isolate the contribution of the curriculum schedule within the knowledge bridge, leaving its specific benefit to the reported gains unresolved.
  2. The cross-task alignment module is described only at a high level, and the generation of pseudo-labels is unclear, so more implementation details is good for reproducibility.
Questions for authors

see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tMvh

We sincerely appreciate the reviewer’s careful and constructive comments. Below, we provide detailed responses to each concern.

W1 The ablation study does not isolate the contribution of the curriculum schedule within the knowledge bridge, leaving its specific benefit to the reported gains unresolved.

Response Thank you for this suggestion. To examine the contribution of the curriculum mechanism within the knowledge bridge, we compare the full method with Without Curriculum, which removes curriculum gating while retaining the remaining components. As shown in Table 1, the full method performs better on four of the five datasets, supporting the contribution of curriculum gating beyond the other components of the knowledge bridge. We additionally evaluate Absolute Curriculum, which retains the gates but removes the subsequent renormalization.

It can be noticed that the variant provides only modest gains over Without Curriculum. A possible explanation is that unnormalized gating reduces both the influence of less reliable sources and the overall transfer strength, potentially weakening useful contributions when gate values are small. Renormalization preserves relative source selection without this additional attenuation. These results support the value of the curriculum mechanism within the knowledge bridge and suggest that selectively weighting sources while preserving transfer strength contributes to its effectiveness.

Table 1 Ablation Study About Curriculum Gate Design

StereoMISSCAREDEndoVis2017EndoVis2018AutoLaparo
Baseline15.2221.390.7030.7180.877
Without Curriculum11.37213.9900.7140.7310.885
Absolute Curriculum11.28113.3500.7240.7300.889
Ours10.3912.520.7290.7260.892

W2 The cross-task alignment module is described only at a high level, and the generation of pseudo-labels is unclear, so more implementation details is good for reproducibility.

Response Thank you for the suggestion. We have added implementation details for the cross-task alignment and pseudo-label generation. At round tt, each client uses its frozen personalized model from round t−1t-1 as the teacher. This teacher has already received cross-task information through the previous bridge aggregation. Given the current client’s local images and prompts, the teacher generates soft predictions for the unlabeled auxiliary task. Specifically, a segmentation client uses the teacher’s depth prediction as a soft depth pseudo-label, while a depth client uses the teacher’s mask probability as a soft segmentation pseudo-label.

For cross-task feature alignment, the student’s primary-task prediction and the teacher’s auxiliary-task prediction are spatially aligned and mapped into a shared embedding space through two-layer projection heads. We then apply an MSE loss between the projected representations. Additional training details are provided in our responses to Reviewer RcUC’s Q2 and W2. We will release our code after the review process to facilitate reproducibility.

Dear Reviewer tMvh,

Thank you again for your thoughtful and constructive review.

We have posted a point-by-point response addressing concerns raised in your review. As the discussion period approaches its conclusion, we would greatly appreciate knowing whether these additional results and clarifications have addressed your concerns, or whether any questions remain. We would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

Thank you for the detailed response, which clarifies my questions. So, I will maintain my positive rating.

Reviewer_srUHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TriEKB-Surg for task-heterogeneous federated learning in surgical video understanding, where clients perform segmentation or depth estimation. The method uses CLIP-based task similarity and a curriculum gate for personalized aggregation, together with teacher-guided cross-task alignment and task-aware feature modulation. Experiments on five public datasets show improvements over existing FL methods.

Critical strengths
  • The paper addresses a practical setting where clients perform different tasks, including segmentation and depth estimation.

  • The which/when/how design provides an intuitive way to organize cross-task knowledge transfer.

  • The experiments include twelve FL and multi-task FL baselines, offering a relatively broad comparison.

Critical weaknesses
  • The cross-task aggregation process is not clearly explained. The paper assumes all clients have identical model dimensions, but their output spaces differ significantly across tasks. It is unclear which parameters are shared and how the aggregation handles different prediction heads.

  • I think the curriculum gate has some design issues. Renormalizing the weights keeps the total transfer strength fixed, which weakens the effect of the gate. Also, the task metrics used, like Dice, depth accuracy, and update cosine, have very different numerical ranges but use the same threshold. This may explain the unusually high cross-task weight at the beginning of training.

  • The teacher is simply the client's previous personalized model, so its predictions on a different task may already be inaccurate. There is no confidence filtering for pseudo-labels, and the feature alignment looks more like regularization than actual cross-task knowledge transfer.

  • The proposition mainly restates a standard descent condition, while the theorem only provides an upper bound. Neither explains the curriculum gate nor establishes convergence or protection against negative transfer. It would be better to reconsider the theorem part and ensure it supports the claims instead of forcing a theoretical part into the manuscript.

Questions for authors

Please refer to my Weaknesses part.

10 comments
AuthorsOpenReview ↗

Response to Reviewer srUH (1/5)

We sincerely appreciate the reviewer’s careful and constructive comments. Below, we provide detailed responses to each concern.

W1 The cross-task aggregation process is not clearly explained. The paper assumes all clients have identical model dimensions, but their output spaces differ significantly across tasks. It is unclear which parameters are shared and how the aggregation handles different prediction heads.

Response Thank you for pointing out this ambiguity. All clients use the same SAM2-Hiera-S architecture and therefore have identical parameter dimensions. The unified decoder contains two structurally identical output branches on every client: a mask branch producing prompt-conditioned binary mask logits and a depth branch producing a single-channel depth map. Segmentation clients optimize the mask output, whereas depth clients optimize the depth output.

Cross-task aggregation is performed in parameter space rather than output space. Specifically, we aggregate only shape-compatible trainable modules, including the mask–depth decoder, memory encoder, memory attention, mask downsampling module, and object-pointer projection. The frozen image encoder is excluded, while task-conditioned or local prompt parameters are retained locally or aggregated only within the same task. Moreover, segmentation is formulated as prompt-conditioned object-wise binary prediction rather than dataset-specific multi-class classification, so different label vocabularies do not introduce incompatible prediction-head dimensions. We will clarify these details in the revised manuscript, and the other implementation details can be found in our Response to Reviewer RcUc.

AuthorsOpenReview ↗

Response to Reviewer srUH (2/5)

W2 I think the curriculum gate has some design issues. Renormalizing the weights keeps the total transfer strength fixed, which weakens the effect of the gate. Also, the task metrics used, like Dice, depth accuracy, and update cosine, have very different numerical ranges but use the same threshold. This may explain the unusually high cross-task weight at the beginning of training.

Response Thank you for raising these concerns. Renormalization is designed to make the curriculum gate determine relative source contributions, while the bridge coefficient α\alpha separately scales the transferred update. Its purpose is therefore to redistribute influence among sources according to their reliability and target compatibility.

Effect of renormalization. As illustrated in Eq. (9), the coarse-level aggregation weight is formulated as:

{\sum_{\ell=1}^{K}g_{i\leftarrow\ell}^{r}\exp(S_{i,\ell}^{c})+\epsilon}.$$ For two sources $j$ and $k$ contributing to the same target $i$ at round $r$, their relative contributions can be expressed as: $$\frac{\widetilde w_{i,j}^{c,r}}{\widetilde w_{i,k}^{c,r}}=\frac{\exp(S_{i,j}^{c})} {\exp(S_{i,k}^{c})} \frac{g_{i\leftarrow j}^{r}} {g_{i\leftarrow k}^{r}}.$$ It shows that the gate directly modifies the semantic contribution ratio through $g_{i\leftarrow j}^{r}/g_{i\leftarrow k}^{r}$. For example, if two sources have equal semantic similarity but gate values of 0.1 and 0.9, their contribution ratio becomes 1:9 rather than 1:1. Renormalization therefore preserves the gate’s ability to suppress a less reliable or less compatible source relative to another source. The denominator converts the gated scores into approximately unit-sum aggregation weights, with the small deviation due to $\epsilon$. This keeps the aggregation coefficient from being additionally scaled by the overall level of the gated scores. In contrast, multiplying semantic weights by gates without renormalization changes both the source mixture and its total weight. Our design assigns these roles separately: $g_{i\leftarrow j}^{r}$ regulates source influence within the mixture, and $\alpha$ in Eq. (11) scales the bridge terms relative to the local update. Importantly, normalizing the weight sum does not fix the transferred-update norm. In Eq. (10), $$\Delta\theta_{i,c}^{r}=\sum_{j=1}^{K}\widetilde w_{i,j}^{c,r}\Delta\theta_j^{r},$$ so changing the relative weights changes how source updates reinforce or cancel one another, as well as which update magnitudes dominate. Consequently, both the direction and norm of $\Delta\theta_{i,c}^{r}$ can change even when the weight sum remains approximately one. Uniformly reducing all gates does preserve their pairwise contribution ratios; the mechanism is therefore primarily relative source selection rather than an absolute switch that turns off all transfer when every source has a small gate. Moreover, we compare our method with Absolute Curriculum, which applies the gates without subsequent renormalization. As shown in Table 1, Absolute Curriculum provides only modest gains over Without Curriculum. A possible explanation is that absolute gating not only downweights less reliable sources but also reduces the overall transfer strength, especially when gate values are small early in training. This can weaken useful contributions and offset the benefit of source selection. Our normalized formulation preserves the relative preference for more reliable sources without introducing this additional attenuation, achieving better performance on most datasets. **Table 1 Ablation Study About Curriculum Gate Design** | | StereoMIS | SCARED | EndoVis2017 | EndoVis2018 | AutoLaparo| | --- | --- | --- | --- | --- | --- | | Baseline | 15.22| 21.39 | 0.703 | 0.718 | 0.877 | Without Curriculum | 11.372|13.990 | 0.714 | **0.731** | 0.885 | | Absolute Curriculum | 11.281|13.350 | 0.724 | 0.730 | 0.889 | | Ours | **10.39** | **12.52** | **0.729** | 0.726 | **0.892** |
AuthorsOpenReview ↗

Response to Reviewer srUH (3/5)

W3 The teacher is simply the client's previous personalized model, so its predictions on a different task may already be inaccurate. There is no confidence filtering for pseudo-labels, and the feature alignment looks more like regularization than actual cross-task knowledge transfer.

Response Thank you for raising this concern. We kindly clarify that our teacher is the previous-round personalized bridged model constructed through Eqs. (9)–(11). It incorporates updates from other clients, including those trained on complementary tasks. The primary role of teacher-guided alignment is to mitigate the effects of heterogeneous training objectives by incorporating complementary-task supervision into local learning. Matching the bridged teacher’s predictions and representations encourages the student to preserve information from other tasks while adapting to its own task.

Specifically, our feature and output alignment serve different but complementary purposes. Feature alignment transfers the representations underlying task prediction, encouraging the student to retain intermediate information learned from complementary-task clients. These representations provide a basis for knowledge transfer beyond the teacher’s final predictions. Output alignment constrains how this information is used, penalizing deviations from the teacher’s complementary-task predictions during local training. Feature alignment therefore supports the transfer of representational knowledge, and output alignment helps preserve its expression at the task level. Together, they encourage the student to retain both the representations acquired through aggregation and the complementary-task behavior they support.

Moreover, the teacher is refreshed after every communication round through personalized knowledge bridging, incorporating newly learned updates from complementary-task clients. It therefore provides a continually updated reference rather than a fixed model that becomes increasingly outdated during local training. Its being from the previous round does not, by itself, imply that it is substantially less capable than the current local model, particularly on complementary tasks for which the client has no ground-truth supervision. This repeated exchange allows the auxiliary supervision to evolve with collaborative training.

We acknowledge that confidence filtering may help reduce the influence of unreliable pseudo-labels. To examine its effect, we evaluate a variant with filtering and another using the previous-round local model as the teacher. As shown in Table 2, our bridged teacher outperforms the local teacher on all five datasets, supporting the value of incorporating cross-client knowledge into teacher supervision. Confidence filtering yields broadly comparable performance. A possible explanation is that filtering removes not only erroneous predictions but also uncertain predictions that still contain useful complementary-task information. Removing these predictions reduces the coverage of the output-alignment constraint, potentially offsetting the benefit of excluding unreliable supervision.

Table 2 Ablation study of teacher selection and pseudo-label confidence filtering

StereoMISSCAREDEndoVis2017EndoVis2018AutoLaparo
Local Teacher11.7214.250.7080.7200.881
w/ Filtering11.0213.050.7010.7350.887
Ours10.3912.520.7290.7260.892
Reviewer_RcUCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks into Federated Learning (FL) in surgical video understanding where it is assumed that different clients tackle different tasks. The core problem of simple FL algorithms like FedAvg is that they do not look into which clients support training and which cause negative transfer. Therefore, the authors claim that we need to ensure that an update from any client is semantically relevant to the target client, that we can only trust the update when the metric improvement is reliable over rounds with an agreement between the target and source update vectors, and the update must be applied to the target client in a consumable fashion by complementary task supervision. To do so, the server has a dual-prompt compatibility module and an evolving curriculum. On the client side, a teacher-student approach is used to transfer knowledge between the tasks. This is supported by a cross-task alignment module. The experiments show that TriEKB-Surg beats state-of-the-art methods and that the combination of the modules made the difference.

Critical strengths
  • Problem setting: In my opinion, the problem setting of task-heterogeneous FL for surgical video is a relevant topic which is sparsely looked into. In particular, the parameter aggregatability does not imply transferability is well motivated.
  • Methodology: The methodology of which, when and how a client update should be processed is explained in detail and reasonably explained.
Critical weaknesses
  • Performance claims: It is reported that TriEKB-Surg beats every state-of-the-art downstream task except for the SCARED dataset. However, there are several questions arising. Firstly, there is no statistical test applied to check if there is a statistical benefit to the runner-up methods or if the benefit lies in the noise of the predictions. Secondly, the results appear to come from a single run and on a mean metric score. It would be helpful to report standard deviations or confidence intervals to understand the variability of the predictions. Furthermore, I noticed that the table has an inconsistency in reporting the best results in bold font. In AutoLaparo the IoU score for FedPer is higher than for TriEKB-Surg. Lastly, the relative improvement over the FedAvg baseline is heavily biased to the MAE score because this score is not bounded between 0 and 1.
  • Reproducibility and central claim: Based on the current explanation I could not reproduce the results, because important hyperparameters with their initialization, model and baseline experiment setups, data splits, and full task prompts are missing. Because of that I am unsure if the central claim of that cross-task collaboration helps is tested or not. Only if the results of Inference and FedAvg report zero shot performance and/or task-wise FedAvg then you have proven that cross-task collaboration helps.
Questions for authors
  • Please provide confidence intervals and statistical tests to clarify if your method really beats the performance or if your improvement is just within the noise of the runner-ups.
  • Please report important hyperparameters with their initialization, model and baseline experiment setups, data splits, and task prompts so that the results are reproducible and understandable.
  • Please explain why a text encoder is needed for creating an unique feature vector for different coarse and fine prompts. In my opinion, a simple one-hot encoded vector is enough and saves computational resources. It could maybe even show a stronger contrast between the different tasks. Could you please discuss this further in your ablation study?
  • Please provide more insights as to what Figure 4 shows exactly. What does 0 and 1 mean?
  • Could you increase the sizes and font sizes of the figures because on printed media they will not be readable?
  • Please note that the EndoVis Challenge has multiple sub-challenges. Please do not only use the EndoVis17/18 challenge dataset name. Please add the complete name (e.g. EndoVis17-RIS dataset)
  • Could you add a discussion and limitation chapter to your work? Please discuss how you anticipate the results are developing when you add more tasks. In particular semantic segmentation and depth estimation are closely related but what about further apart tasks.
8 comments
AuthorsOpenReview ↗

Response to Reviewer RcUC (1/5)

We sincerely appreciate the reviewer’s careful and constructive comments. Below, we provide detailed responses to each concern.

W1.1 and Q1 Please report standard deviations or confidence intervals and assess whether improvements over the strongest competing methods are statistically significant.

Response Thank you for your comment. We use a shared training and evaluation pipeline across all methods to ensure a fair comparison. Given the limited rebuttal period and computational resources, we repeated experiments for our method and three strong competing methods, FedAWA, FUSS, and FedHCA2, using three random seeds. We evaluate depth estimation and segmentation using MAE and Dice, respectively.

Table 1 reports the mean and sample standard deviation. Our method achieves the best mean performance on four of the five datasets. To assess statistical significance, we conducted two-sided Welch’s t-tests using the individual run results. Before multiple-comparison correction, our method significantly outperforms FedAWA and FUSS on StereoMIS, and FUSS and FedHCA2 on EndoVis2017 (p<0.05p<0.05). On AutoLaparo, FUSS already achieves performance close to ours, and the small difference in mean Dice is not statistically significant. These results support the benefits of our method across heterogeneous tasks.

Table 1 Performance across three random seeds (mean ± sample standard deviation).

StereoMISSCAREDEndoVis2017EndoVis2018AutoLaparo
FedAWA14.30±1.2014.30 \pm 1.2012.43±6.1912.43 \pm 6.1960.53±7.5460.53 \pm 7.5461.97±9.4061.97 \pm 9.4086.60±1.6186.60 \pm 1.61
FUSS11.79±0.4011.79 \pm 0.4012.09±0.6012.09 \pm 0.6068.30±0.6268.30 \pm 0.6271.83±0.5571.83 \pm 0.5588.80±0.1088.80 \pm 0.10
FedHCA211.77±0.7611.77 \pm 0.7612.41±0.3012.41 \pm 0.3069.30±0.8969.30 \pm 0.8971.77±0.5571.77 \pm 0.5588.53±0.2188.53 \pm 0.21
Ours10.71±0.4910.71 \pm 0.4912.37±0.1312.37 \pm 0.1372.10±0.7272.10 \pm 0.7272.80±0.2072.80 \pm 0.2088.87±0.4988.87 \pm 0.49

Table 2. Two-sided Welch’s t-tests comparing our method with each baseline.

ComparisonStereoMISSCAREDEndoVis2017EndoVis2018AutoLaparo
Ours vs. FedAWA0.02240.98810.11600.18400.1249
Ours vs. FUSS0.04550.50030.00250.07950.8384
Ours vs. FedHCA20.12500.83470.01450.06930.3680

W1.2 Performance reporting: Please correct the best-result highlighting and performance claims, as FedPer achieves higher AutoLaparo IoU. Also clarify whether the aggregate relative improvement is dominated by MAE and adequately reflects performance across tasks.

Response Thank you for your comment. We acknowledge that FedPer achieves the highest AutoLaparo IoU and have corrected the highlighting and corresponding performance claims. We acknowledge that our method does not outperform all competing methods on every individual metric, but achieve a satisfactory tradeoff among different tasks.

Moreover, we thank the reviewer that notice that the overall relative improvement can be disproportionately influenced by depth estimation gains. Hence, to make this distinction explicit, we report task-specific relative improvements separately: 34.49% for depth estimation and 3.26% for segmentation, both relative to FedAvg. Our backbone is pretrained for segmentation, providing a stronger starting point for segmentation than for depth estimation and leaving less room for relative improvement on segmentation. Furthermore, the relative improvement calculation depends on the baseline value: reductions in depth estimation errors can yield substantially larger percentage improvements than gains in segment tasks.

Table 3 Task-wise and average relative improvements over FedAvg (%).

MethodΔm Depth (%) ↑Δm Seg (%) ↑Average Δm (%) ↑
Inference-359.63-13.32-186.48
FedAvg0.000.000.00
FedProx-37.09-17.66-27.38
FedProx*19.73-11.973.88
FedAWA28.75-19.224.77
FUSS30.540.9515.75
FedBN1.82-1.140.34
LoRA-Fair-149.330.79-74.27
FedNova-19.25-6.04-12.65
FedPer-73.37-1.38-37.37
FedHCA232.231.2616.74
MaTFL25.931.2013.57
HNFL28.800.4614.63
Ours34.493.2618.87
AuthorsOpenReview ↗

Response to Reviewer RcUC (2/5)

W2 and Q2 Please report important hyperparameters with their initialization, model and baseline experiment setups, data splits, and task prompts so that the results are reproducible and understandable.

Response: Thanks for the suggestion, and we will add a table in the supplmentation to include the implementation details.

Model and initialization. All methods use the pretrained SAM2-Hiera-S as the backbone, with the image encoder frozen. The pretrained CLIP ViT-B/32 text encoder is also kept frozen. Newly introduced learnable parameters in the prompt-conditioning module are initialized from a zero-mean Gaussian distribution with a standard deviation of 0.02.

Optimization. We train all methods for 100 communication rounds, with one local epoch per round, a batch size of 1, and an input resolution of 512×512512\times512. Decoder-related modules are optimized using Adam with a learning rate of (10−410^{-4}). The previous-round consistency weight is 0.05. The task/global bridge coefficients are both 0.3, and the prompt-similarity temperature is 0.1. The curriculum uses a 30-round warm-up period and an exponential moving average (EMA) coefficient of 0.8 for reliability estimation.

Baseline Setup. All controlled baselines use the same pretrained model, frozen/trainable modules, data splits, optimizer, local epochs, communication rounds, and evaluation budget.

Complete Task Prompts. Our method uses two types of textual prompts for each client: a coarse task-family prompt and a fine task-semantic prompt. Across the five clients, this gives ten prompt assignments and six distinct prompt strings, comprising two coarse prompts and four fine prompts. The complete prompts are as follows:

Coarse Task-Family Prompt:

  • Depth-estimation clients (StereoMIS and SCARED): "Task: depth estimation"

  • Segmentation clients (EndoVis17, EndoVis18, and AutoLaparo): "Task: surgical image segmentation"

Fine Prompt:

  • StereoMIS: "Estimate dense pixel-wise depth from endoscopic surgical images"
  • SCARED: "Estimate dense pixel-wise depth from endoscopic surgical images"
  • EndoVis17: "Segment surgical instrument parts, including shaft, wrist, and clasper"
  • EndoVis18: "Segment surgical instruments and anatomical structures, including kidney, intestine, suction instrument, ultrasound probe, and surgical tools"
  • AutoLaparo: "Segment surgical instruments and anatomical structures in laparoscopic surgery"

The coarse prompt defines broad task-family compatibility over all clients, whereas the fine prompt refines compatibility within a task family when more specific task semantics are available. StereoMIS and SCARED share the same fine prompt because both perform the same dense depth-estimation task; we deliberately avoid encoding dataset identity as task semantics. Their contributions are subsequently differentiated by the evolutionary curriculum using source reliability and target-specific update agreement. In contrast, the fine prompts distinguish the three segmentation clients because they involve different segmentation targets.

Moreover, we will release our code to facilitate reproducibility and provide implementation details.

AuthorsOpenReview ↗

Response to Reviewer RcUC (3/5)

W2 and Q3 Please explain why a text encoder is needed for creating an unique feature vector for different coarse and fine prompts. In my opinion, a simple one-hot encoded vector is enough and saves computational resources. It could maybe even show a stronger contrast between the different tasks. Could you please discuss this further in your ablation study?

Response Thank you for this insightful question. Although the client tasks can be broadly categorized as either segmentation or depth estimation, their fine-grained objectives are neither identical nor completely unrelated. Importantly, the relationships among these tasks are not strictly orthogonal, as implicitly assumed by one-hot encoding. For example, EndoVis17 focuses on segmenting surgical instrument parts, whereas EndoVis18 and AutoLaparo involve the segmentation of both surgical instruments and anatomical structures. These tasks share substantial semantic content while differing in their specific target categories. Manually defining such pairwise relationships would introduce subjectivity and may fail to capture meaningful dependencies. We therefore use textual task descriptions to estimate client compatibility from their semantic information.

As mentioned earlier, one-hot encoding assigns orthogonal vectors to different task labels and consequently treats every pair of distinct labels as equally dissimilar. It therefore cannot represent the graded semantic relationships described above. In contrast, our method employs a frozen text encoder to map the prompts into a shared semantic space, allowing semantically related tasks to exhibit higher compatibility than less-related ones. Furthermore, we introduce a dual-level prompt design. The coarse prompt captures broad task-family compatibility across all clients, such as the distinction and potential relationship between segmentation and depth estimation. The fine prompt represents more specific task semantics and refines compatibility within a task family when fine-grained differences are present. For instance, it distinguishes instrument-part segmentation from the joint segmentation of instruments and anatomical structures.

To facilitate this comparison, we conduct an additional ablation study, with the results reported in Table 4. We replace our prompt design with one-hot encoding while keeping all other components and experimental settings unchanged. Although one-hot encoding achieves competitive performance, our prompt design yields better results on four of the five datasets, with particularly notable improvements on SCARED and EndoVis2017. One-hot encoding performs slightly better on AutoLaparo. Overall, these results suggest that modeling latent relationships among clients improves performance.

Table 4 Ablation Study About Prompt Design

StereoMISSCAREDEndoVis2017EndoVis2018AutoLaparo
Baseline15.2221.390.7030.7180.877
Random Orth10.9117.350.6960.7240.878
One-Hot10.6316.770.6830.7120.900
Ours10.3912.520.7290.7260.892

Regarding computational efficiency, we acknowledge that the text encoder introduces additional computation. However, this cost is incurred only once before federated training, when the frozen encoder generates the prompt embeddings. These embeddings are shared across all samples, cached, and reused throughout all communication rounds. Thus, text encoding incurs a one-time preprocessing cost rather than recurring overhead during training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.