acceptodds
Withdrawn from ICLR 2027

QGA-TTT: Query-Guided Aggregation via Test-Time Training for Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical image segmentation asks for a label at every position of a scene that holds many structures at once, and which class a region belongs to is decided in large part by what else the image contains. Every position of one image therefore has to be labelled against one and the same account of that image. Convolution reaches only as far as its receptive field, and pairwise attention lets positions compare with one another without producing a single shared description, so neither forms such an account. Test-time training layers fit a small set of fast weights inside the forward pass, so the map they apply is recomputed for every input. We use this property to build one shared account per image. At each of the deepest levels of a segmentation network, the level’s positions together with a small set of learnable queries shared across every image are fitted, in a single forward pass, into one affine map, and every position reads the global information back through that same map. The shared queries enter every fitting, so decoder weights shared across images can read what the map returns. The queries are kept across two passes so that the second reads an account already placed in the coordinates of the current image, and the map is estimated twice by fittings that share no parameters. Across nine public datasets spanning endoscopy, ultrasound, optical coherence tomography and magnetic resonance imaging, the method reaches the best Dice on every one, by up to 2.81 points over the strongest comparison method. Ablations show each of the three constructions to be necessary, and both the number of fittings and the capacity of the account are interior optima.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image segmentation, test-time training, global context, learnable queries

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_YbRQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes QGA-TTT, a global feature aggregation module for medical image segmentation built on test-time training layers. At each of the three deepest skip connections of a convolutional encoder–decoder, spatial features are concatenated with learned queries shared across images. A single gradient step fits head-wise affine fast weights from this working set, and all positions read information through the fitted maps. The module retains the query representations across two successive blocks, averages two separately parameterized fitting branches within each block, and injects the resulting correction through a learnable channel-wise gate. The paper evaluates the method on nine public datasets, including six 2D and three volumetric segmentation tasks. It reports the highest Dice score among the evaluated methods on every dataset. Additional experiments examine component ablations, computational cost, boundary accuracy, fitted-map spectra, and distances between maps fitted to different images.

Critical strengths

The broad evaluation covers nine heterogeneous datasets and includes convolutional, attention-based, state-space, and existing TTT architectures.

Critical weaknesses
  1. Limited novelty. The main components (including fitting weights within the forward pass, inner learner, and auxiliary tokens) have precedents in existing TTT and Transformer literature. The contribution beyond combining these established designs is insufficiently elaborated.

  2. In the 2D setting, QGA-TTT nearly doubles nnU-Net’s parameter count (87.3M vs. 46.3M). It remains unclear how much improvement comes from the proposed TTT mechanism versus increased model capacity.

  3. Table 3 covers only two datasets, providing limited support to analyze the contribution of each component. Moreover, carrying queries across blocks reduces performance on both datasets when paired fitting is disabled, and this degradation is lack of explanation.

  4. The Table 3 baseline, with all three listed components disabled, outperforms all competing methods shown in Table 1 on both m2caiSeg and Endoscapes datasets. It raises concerns about comparison fairness and the attribution of gains.

  5. Overlapping markers and curves in Figure 4 obscure individual results and make the intended comparisons difficult to interpret.

  6. The writing is poor. The manuscript requires substantial rewriting to improve clarity and precision.

Questions for authors
  1. What is the substantive methodological advance over prior TTT methods, beyond combining established components?

  2. Which components account for the parameter increase from 46.3M to 87.3M? Can you provide a parameter-matched comparison with nnU-Net to separate the benefits of the proposed TTT mechanism from those of increased capacity?

  3. Can you provide ablations on additional datasets to assess whether the claimed component benefits generalize?

  4. Why does carrying queries across blocks reduce performance when paired fitting is disabled?

  5. Why the baseline shown in Table 3 outperforms all competing methods on both m2caiSeg and Endoscapes datasets?

Reviewer_XVJtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QGA-TTT, which fits an image-specific map during the forward pass to aggregate global information for medical image segmentation. The method achieves strong results across multiple datasets, but its inference cost, the stability of its smaller performance gains, and the interpretation of its mechanism require further clarification.

Critical strengths
  1. The method has a clear motivation. It addresses how the structure of an entire image can inform classification at individual positions through a feature aggregation mechanism that adapts to each input image.
  2. The evaluation covers a broad range of settings. The method is compared with convolutional, attention-based, state-space, and test-time-training approaches on nine datasets spanning endoscopy, ultrasound, OCT, and MRI.
Critical weaknesses
  1. The explanation of the core mechanism needs more direct evidence.
    • The evidence for a “shared image account” remains indirect. The low effective rank of the fitted maps and the reduced distances between maps from different images do not, by themselves, show that the maps encode image structure useful for segmentation. An analysis linking the fitted maps to segmentation behavior would strengthen this interpretation.
    • The mechanism of information transfer between passes needs clarification. The authors describe the queries as the only route by which the image account formed in the first pass reaches the second. However, according to Eq. (4), the entire updated working set U(1), including both spatial and query tokens, is passed to the second block. The authors should explain why the updated spatial tokens cannot carry the relevant information and what distinct role retaining the queries plays in transferring information between passes.
  2. The performance gains and the contributions of individual components need further validation.
    • Inference is relatively expensive. The reported single-case forward-pass time is approximately 4.3–7.2 times that of nnU-Net. The paper should discuss this cost relative to the Dice improvements, particularly on datasets where the gains are small.
    • The ablations cover a limited range of settings. Although the method is evaluated on nine datasets, component ablations are conducted on only two multiclass 2D datasets. It is therefore unclear whether shared queries and paired fitting are equally beneficial on datasets with fewer classes or in 3D segmentation tasks.
Questions for authors

I would appreciate clarification on the weaknesses raised above. If these concerns are adequately addressed, I would be willing to consider raising my rating during the rebuttal period.

Reviewer_VV8aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes QGA-TTT, a medical image segmentation framework that uses test-time training to fit image-specific linear models from spatial features and shared learnable queries, with iterative query refinement and paired fast-weight fitting to enhance global feature aggregation. Experiments on nine medical imaging datasets report consistently higher Dice scores than existing methods, although the approach incurs substantial computational overhead and the motivation for its self-supervised TTT objective remains unclear.

Critical strengths
  1. Extensive experimental evaluation. The paper evaluates QGA-TTT on nine medical image segmentation datasets spanning four imaging modalities and compares it against a large collection of existing methods, including TTT-based architectures. The consistently strong Dice scores provide empirical evidence for the effectiveness of the proposed architecture.

  2. Reasonably comprehensive ablation studies. The authors examine the contributions of the learnable queries, iterative query refinement, and paired fast-weight fitting. They also investigate the effects of the number of fittings, fast-weight capacity, and the number of enhanced feature levels. These experiments help identify which architectural choices contribute to the reported improvements.

Critical weaknesses
  1. The primary issue of the paper is that the terminology is unusually unnatural, the explanations unnecessarily convoluted, and the central argument difficult to follow. These are signs of LLM writing without sufficient human polishing. Here is a table of representative examples:
Expression in the paperMore natural technical terminology
"one and the same account of that image"Shared global representation
"the level's own positions"Spatial feature tokens
"the material the fit reads"Input features
"queries that have already been placed"Updated query embeddings
"the working set"Concatenated feature tokens and queries
"the number of fittings"Number of parallel fast-weight updates
"an account already placed in the coordinates of the current image"Image-conditioned representation

In particular, the authors use "account" repeatedly throughout the abstract, introduction, methodology, and conclusion. It becomes an invented conceptual framework for describing familiar operations involving global representations and fast weights.

  1. Unclear motivation for the self-supervised TTT objective (Eq.2). The paper does not adequately explain why the proposed feature reconstruction objective is particularly suitable for medical image segmentation. Since both the key and value features are learned projections of the same input tokens, the outer segmentation objective can optimize these projections to produce useful fast-weight updates without necessarily encouraging meaningful feature reconstruction. In this sense, the inner objective may simply serve as a mechanism for generating input-dependent weight updates, rather than as a meaningful self-supervised learning task. The authors should clarify the rationale behind this objective and investigate whether alternative objectives or simpler input-dependent weight generation mechanisms achieve comparable performance.
Questions for authors

N/A

Reviewer_kzTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QGA-TTT adds global-context modules to the three deepest skip connections of a residual convolutional encoder–decoder. Each module combines spatial features with learned queries shared across images. Within each fitting branch, one gradient step produces an input-dependent affine map through which all tokens read information. The queries are retained across two sequential blocks, each averaging the outputs of two independently parameterized fitting branches. A channel-wise gate adds the resulting correction to the skip connection. The method is evaluated on nine datasets covering four imaging modalities and both 2D and volumetric segmentation. It achieves the highest reported foreground-averaged Dice on all nine datasets, with gains of 0.16–2.81 percentage points over the strongest comparator on each dataset. The paper also reports HD95, computational costs, ablations, and analyses of the fitted maps.

Critical strengths
  1. Fitting a map from an entire feature level, together with shared queries, is an interesting use of fast weights for global aggregation. Retaining the queries across blocks gives them a way to carry image-dependent information into the next fit. The underlying TTT mechanism is established, but this particular design is worth studying.

  2. The method ranks first in Dice on all nine datasets, including improvements of 2.81, 1.75, and 2.37 points on OCT5k, m2caiSeg, and Endoscapes. The evaluation covers several modalities, 2D and 3D tasks, and existing TTT-based methods. The authors also report boundary metrics and actual computational costs, allowing readers to assess the trade-offs.

Critical weaknesses
  1. Table 4 reports 52.25/55.55 Dice for “nothing, gate at zero” on m2caiSeg/Endoscapes, while nnU-Net in Table 1 scores 50.66/53.54. The complete model scores 53.25/56.34. If the zero-gate model was trained independently with the correction disabled, the gains over this internal baseline are 1.00/0.79 points. If the gate was set to zero after training, the result measures a different intervention. I could not find an explanation of which experiment was performed or why the two baselines differ.

  2. Table 3 also lacks a clear comparison isolating query carry-over. The row retaining queries but removing carry-over also removes paired fitting. Another row marks carry-over as present without queries. These configurations need clarification before the table can support the necessity of all three components. Repeated-run uncertainty would also help establish whether the smaller reported gains are consistent.

  3. Non-causal linear attention already forms global statistics shared across positions (Katharopoulos et al., 2020), so shared global readout alone does not distinguish QGA-TTT. The contribution lies in how the map is fitted and how queries affect that fit. The “linear aggregate” control in Table 4 is relevant, but its formula and matching conditions are not specified.

  4. Section 3.3 also states that queries are the only route carrying the first pass’s account into the second. However, Equations (4)–(5) pass the updated spatial tokens into the second block as well. Those tokens can also carry information from the first fit.

  5. The smaller cross-image map distances and concentrated singular-value spectra are useful observations, but they do not by themselves establish semantic coordinate alignment or show why the decoder needs it. I would find the paper more convincing with a clean query-persistence ablation and a clearer distinction between measured behavior and the authors’ interpretation.

Questions for authors
  1. Was “nothing, gate at zero” trained independently with a permanently zero gate, or obtained by modifying a trained complete model? What explains its difference from nnU-Net in Table 1, and which baseline matches the full model’s backbone and training configuration?
  2. Which row in Table 3 removes only query carry-over while retaining queries and paired fitting? What is carried over when queries are absent? If the isolated control is missing, could you report a query-reset experiment with the spatial-token path and other settings unchanged? Please also define the “linear aggregate” control and explain how its parameters and computation are matched.
  3. Please clarify the dataset-specific split sizes, patient/procedure separation, and checkpoint and architecture selection rules. Were the reported evaluation cases used for model selection? Are repeated-seed or additional-fold results available on representative datasets? Since Section 5 describes consecutive frames from one procedure in the m2caiSeg validation set, it would help to distinguish variation across frames from generalization across procedures.
  4. What inner-loop step size, loss reduction, normalization axes, and gradient treatment are used? How does the “only route” statement account for the updated spatial tokens entering the second block? Is there evidence that separates semantic alignment from a shared prior or reduced input sensitivity, or should this interpretation be treated as a hypothesis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.