acceptodds
Under review as a conference paper at ICLR 2027

SRTAR: Super-Resolution Injection with Task-Aware Refinement for Challenging Object Segmentation

Abstract

Efficient extraction of high-quality features from input images is critical for accurate challenging object segmentation. Real-world inputs often contain low-resolution (LR) contents, in this circumstance, even employing transformer architectures providing powerful feature modeling, achieving satisfactory performance is limited by the lack of high-frequency components crucial for the task within LR content. Image super-resolution (SR) is one of the promising solutions. However, due to the ill-posed property of SR, typical SR methods struggle to restore task-relevant high-frequency contents, which may dilute the advantage of utilizing the SR method. Therefore, we propose Super-Resolution Injection with Task-Aware Refinement (SRTAR) that effectively guides the generation of SR images beneficial to achieving satisfactory segmentation performance when processing LR images. The critical component is task-aware refinement (TAR) that enables the SR branch to acquire task-relevant discriminative knowledge from segmentation tasks and provides detail-rich features for segmentation refinement. Moreover, we propose a mixed-quality patch stack and a trilateral fusion loss to enhance the efficacy of TAR by addressing potential problems when employing the TAR. Extensive experiments demonstrate that our SRTAR achieves outstanding performance by generating SR images useful for specific task across 8 challenging object segmentation tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Binary Object Segmentation, Image Super-Resolution, Task-Aware Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kt4kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Super-Resolution Injection with Task-Aware Refinement (SRTAR) that effectively guides the generation of SR images beneficial to achieving satisfactory segmentation performance when processing LR images. The critical component is task-aware refinement (TAR) that enables the SR branch to acquire task-relevant discriminative knowledge from segmentation tasks and provides detail-rich features for segmentation refinement. Moreover, it proposes a mixed-quality patch stack and a trilateral fusion loss to enhance the efficacy of TAR by addressing potential problems when employing the TAR. Extensive experiments demonstrate that SRTAR achieves outstanding performance by generating SR images useful for specific task across 8 challenging object segmentation tasks.

Critical strengths

SRTAR achieves outstanding performance by generating SR images useful for specific task across 8 challenging object segmentation tasks.

Critical weaknesses

The figures in this papers are blurred; the clear figures should be presented.

The metrics should be introduced, such as S_m, F^w_beta, M, Dice. Why are there no quantitative comparisons for SwinIR, while qualitative comparisons include its results? Please explain.

SwinIR is too old for image super-resolution, the recent methods should be compared and discussed. Such as [1] Dual-Level Cross-Modality Neural Architecture Search for Guided Image Super-Resolution. TPAMI [2] Progressive Focused Transformer for Single Image Super-Resolution. CVPR [3] DiT4SR: Taming Diffusion Transformer for Real-World Image Super-Resolution. ICCV [4] CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution. CVPR [5] Uncertainty-guided Perturbation for Image Super-Resolution Diffusion Model. CVPR Meanwhile, some recent object segment methods should be compared and discussed, such as: [1] Semantic and sequential alignment for referring video object segmentation. CVPR [2] Referdino: Referring video object segmentation with visual grounding foundations. ICCV [3] Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation. CVPR [4] Lvos: A benchmark for large-scale long-term video object segmentation. TPAMI [5] Refsam: Efficiently adapting segmenting anything model for referring video object segmentation. AAAI

Questions for authors

Please view Critical Weaknesses

Reviewer_NdhdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SRTAR, a task-aware super-resolution framework for low-resolution binary object segmentation. The main idea is to inject SR information into multiple segmentation decoder stages rather than using SR only as an image preprocessing step. The framework consists of three components: Task-Aware Refinement (TAR) for multi-stage SR feature injection and gradient-guided refinement, Mixed-Quality Patch Stack (MQPS) for combining HR and SR image patches at different scales, and Trilateral Fusion Loss (TFL) for improving pixel, structural, and gradient consistency. The method is evaluated on eight binary segmentation tasks and shows improvements over conventional SR-based segmentation baselines.

Critical strengths

The paper addresses an important problem in low-resolution object segmentation. The motivation is reasonable, as improving image reconstruction quality does not necessarily improve downstream segmentation performance. The evaluation across eight segmentation tasks under two resolution settings provides useful evidence of the applicability of the proposed framework.

The proposed method achieves encouraging improvements over conventional SR-based segmentation baselines. The ablation studies also show the benefits of the proposed components. In particular, SRTAR_B achieves performance close to the full SRTAR while reducing computational costs compared with SRTAR, which is an interesting result.

Critical weaknesses
  1. Limited methodological novelty and unclear source of performance improvements.

The proposed framework mainly integrates existing techniques, including multi-scale feature fusion, deformable convolution, gradient-guided attention, mixed-quality image augmentation, and structural consistency losses. Although the integration is reasonable, the technical novelty is not sufficiently clear. More importantly, the performance difference between SRTAR_B and SRTAR is very small, even though SRTAR_B replaces SwinIR with bilinear upsampling during inference. This raises an important question about the actual source of the performance improvement. It is unclear whether the gains mainly come from the reconstructed SR information, the increased network complexity, the proposed refinement modules, or mixed-quality image injection, which may work similarly to data augmentation. The current experiments do not sufficiently distinguish these factors, weakening the central motivation and contribution of the paper.

  1. Insufficient analysis of computational cost and practical efficiency. The proposed approach processes super-resolved images instead of the original LR inputs, which can substantially increase the computational cost of the segmentation network. For example, ×2 super-resolution increases the number of input pixels by four times, potentially resulting in much higher computational and memory requirements.

Although the paper compares the computational costs of SRTAR and SRTAR_B, a comprehensive comparison with the original LR-based segmentation baseline and conventional SR-based methods is missing. It is therefore difficult to determine whether the segmentation improvements justify the additional computational cost. This is particularly important because computational efficiency is a key consideration for low-resolution image applications.

Questions for authors
  1. What is the main source of the performance improvement? Since SRTAR_B achieves performance close to SRTAR without performing SR reconstruction during inference, could the authors clarify the actual contribution of SR information? It would be useful to compare the proposed method with a network of similar complexity using bilinear-upsampled images and the same refinement modules. In addition, could the improvements from MQPS be mainly attributed to mixed-quality data augmentation rather than the proposed stage-matched refinement strategy?

  2. How does the computational cost compare with the original LR-based segmentation? Could the authors provide FLOPs, inference time, and GPU memory usage for all main baselines, including direct LR segmentation, bilinear upsampling, conventional SR-based segmentation, SRTAR_B, and SRTAR? This comparison is important to assess the trade-off between segmentation accuracy and computational efficiency.

  3. How are the gradient-based losses implemented? BCE loss is applied to gradient maps. Are these maps normalized to [0, 1], and how are negative or large gradient values handled? Also, how is the task-relevant weighting map constrained to avoid trivial solutions where the gradient responses are suppressed instead of being properly aligned?

  4. How are the mixed-quality patches constructed and aligned? Could the authors clarify the patch sizes, grid configurations, and spatial alignment operations used in MQPS? Since mixed HR/SR patches are used during training while only SR patches are used during inference, how does this training-inference difference affect the performance?

Reviewer_btgLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SRTAR targets binary object segmentation from low-resolution inputs. Instead of using SR as a frozen preprocessor, it couples a SwinIR SR branch with a Swin-based segmentation decoder. The core module, Task-Aware Refinement (TAR), injects stage-matched SR/patched image cues into multiple decoder stages and applies a gradient-alignment loss so that the SR output mimics HR gradient responses in task-relevant regions. To avoid shortcut learning, Mixed-Quality Patch Stack (MQPS) builds HR/SR mixed patches per decoder stage, and Trilateral Fusion Loss (TFL) aligns SR and segmentation representations at pixel, structural-affinity, and gradient levels. The paper evaluates eight tasks (COD, SOD, polyp, COVID lung, breast lesion, skin lesion, transparent objects, defocus blur) under ×2/×4 bicubic degradation, and proposes SRTAR_B, a bilinear-inference variant that keeps most gains without running SwinIR at test time.

Critical strengths
  1. Problem formulation is practical.​ LR segmentation with degraded boundaries/small structures is a real bottleneck; the paper correctly argues that pixel-fidelity SR does not automatically help downstream segmentation.
  2. Stage-wise injection is reasonable.​ Feeding SR cues into multiple decoder stages rather than only at input level helps counteract downsampling loss in the encoder—this is the most useful design idea here.
  3. MQPS is well-motivated.​ The shortcut-learning argument is sound, and patch-level HR/SR mixing is a sensible regularizer to stop the segmenter from over-relying on SR artifacts.
  4. TFL gives explicit cross-branch alignment.​ Combining pixel consistency, affinity similarity, and gradient BCE is a reasonable way to bind the SR branch to task regions, even if not highly novel.
  5. SRTAR_B is a genuinely useful angle.​ If most gain survives without test-time SR, that matters for deployment. The FLOPs/latency table makes the paper more than a pure accuracy claim.
  6. Broad evaluation scope.​ Eight heterogeneous binary-seg tasks is more than typical SR-for-segmentation papers and gives a decent generality signal》
Critical weaknesses
  1. Novelty is incremental and prior art is under-cited. (1)Task-driven SR is already established: Haris et al. (task-driven SR for detection), Wang et al. (dual SR learning for segmentation), and especially Kim et al. SR4IR with task-feature perceptual loss + cross-quality patch mix. (2)MQPS is essentially CQMix applied at decoder-stage patches. (3)Gradient/boundary alignment losses are common in segmentation. (4)The paper should explicitly position TAR as decoder-stage injection for binary segmentation, not imply the task-aware SR idea is new.
  2. Unfair comparison with I_S → T.​ In I_S → T, SwinIR is pretrained then frozen and only the segmenter trains on SR outputs. In SRTAR, SwinIR is fine-tuned end-to-end with segmentation losses. That is a different capacity regime. The gain over I_S → T therefore conflates “better SR for the task” with “extra trainable capacity + joint optimization.” A fairer baseline is needed: freeze SwinIR weights but allow TAR/MQPS/TFL, or fine-tune SwinIR with segmentation loss without TAR.
  3. TAR gradient loss uses I_SR gradient, not prediction gradient.​ Eq. (1) aligns SR gradient with HR gradient inside a task-weighted map. But the segmentation boundary improvement should arguably come from aligning predicted mask gradient with GT boundary, as in standard boundary losses. As written, TAR supervises the SR branch’s image gradients; the connection to final mask boundaries is indirect and undemonstrated.
  4. Only bicubic degradation.​ Real old photos, surveillance, medical scanners, or tiny-object LR are not bicubic. The claim that SRTAR helps “real-world LR” is currently synthetic-only. At minimum, one real-degradation or blur+downsample ablation should be added.
  5. SRTAR_B needs a controlled ablation.​ If SRTAR_B replaces SwinIR with bilinear only at inference, its gain may come from training on MQPS mixed patches rather than from TAR inference logic. The paper should report: (1)SRTAR_B trained without MQPS, (2)bilinear baseline trained with the same segmenter,to isolate why the variant retains performance.
  6. SR quality claim is thin.​ Table 3 claims “best PSNR/LPIPS” on SOD, but task-driven SR often trades PSNR for task utility. Showing only one task is not enough to claim “visually appealing SR” broadly.
Questions for authors
  1. Prior art scope.​ Why is SR4IR / task-driven SR4IR not discussed in Related Work? What exactly is the new mechanism beyond “SR4IR + decoder injection”?
  2. Fair baseline.​ Can you add an I_S → T variant where SwinIR is fine-tuned with the segmentation loss but without TAR/MQPS? Otherwise the gain over sequential SR is confounded by joint training.
  3. Ablation isolation.​ Please provide drop-ablations for TAR / MQPS / TFL separately, ideally on COD + one medical task. How much does each component contribute at ×4?
  4. Gradient-loss design.​ Why supervise I_SR gradient rather than predicted-mask gradient? Did you test boundary loss on the segmentation output directly?
  5. MQPS at inference.​ During inference MQPS uses only I_SR; doesn’t this lose the HR-patch regularization that stabilizes training? Does SRTAR_B behave differently if MQPS is disabled at test time?
  6. SRTAR_B attribution.​ Is the retained accuracy from TAR or simply from having seen SR/HR mixed inputs during training? Please compare SRTAR_B against a bilinear baseline trained with the same augmentation.
  7. Evaluation fairness.​ In Table 1, several SOTAs use larger inputs/architectures. Can you re-evaluate the strongest 2–3 baselines at the same 256/512 protocol?
  8. Real degradation.​ Bicubic ×4 is convenient but weak. Have you tested Gaussian blur + noise + JPEG, or any real LR medical/surveillance source?
  9. SR metrics.​ Table 3 alone does not support “SR images are visually pleasing.” Can you report PSNR/SSIM/LPIPS on all tasks, and maybe a user study on SR realism vs task usefulness?
  10. Complexity vs gain.​ On ×2, SRTAR often beats I_S→T by small margins. Is the added TAR/TFL overhead worth it outside ×4? Please comment on when SRTAR actually matters.
1 comment

The author did not respond to my question, so I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.