acceptodds
Under review as a conference paper at ICLR 2027

Pixel-Level Tasks Need State-Aware Compensation and Adaptation

Abstract

Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of pre-trained models to downstream tasks by updating only a small subset of task-specific parameters, but still requires full model computation during inference. Although token reduction (TR) improves inference efficiency by assigning different computational paths to tokens, it introduces partial information loss as a consequence of reduced token computation. This challenge is even more pronounced in pixel-level tasks, which require dense and fine-grained spatial information for accurate prediction. Existing TR-PEFT methods address this challenge by increasing adaptation capacity or introducing explicit compensation pathways. However, these designs typically do not explicitly account for the distinct token states arising from TR-induced computational paths, resulting in a state-agnostic formulation. Our analysis reveals a design gap in state-agnostic approaches under TR, with observable effects on compensation effectiveness and adaptation differentiation across tokens. To address this gap, we propose a state-aware approach that leverages TR-induced token states to regulate compensation and adaptation accordingly. Our approach effectively improves compensation and adaptation, achieving better performance with reduced computational overhead across diverse pixel-level tasks. We further demonstrate competitive or superior performance compared with strong baseline methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Parameter-Efficient Fine-Tuning, Token Reduction, Pixel-Level Tasks, State-Aware

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HvhUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the integration of parameter-efficient fine-tuning and token reduction for pixel-level tasks. The authors point out that existing TR-PEFT methods are state-agnostic to the two distinct token states induced by TR (preserved and skipped tokens). To tackle this issue, they propose TRACE, a state-aware design that leverages TR-induced token states to regulate feature compensation and adaptation. Experimental results demonstrate that TRACE consistently outperforms existing approaches across multiple dense task benchmarks.

Critical strengths
  1. The problem analysis of existing TR-PEFT approaches is well-conducted; characterizing the behavioral anomalies of state-agnostic designs from three measurable perspectives provides valuable insights for advancing the TR-PEFT direction.

  2. The proposed method is concisely designed. The core mechanism lies in explicitly propagating the pre-existing selection signals from the Token Selector to the compensation mask and the adaptation-compensation mixture (Eqs. 10–12). It introduces virtually no complicated paradigms while keeping both trainable parameters and FLOPs strictly low.

Critical weaknesses
  1. While the paper claims to “achieve better performance with reduced computational overhead”, the actual improvements over the baseline (DAR) are marginal and inconsistent, mostly hovering in the 0.005–0.01 range. Crucially, all reported results appear to stem from a single run without providing mean ± standard deviation or any statistical significance testing, making it impossible to rule out random seed variance. Furthermore, there are multiple notable performance degeneration: e.g., AdaptFormer w/o-TR outperforms both Ours and Ours^SD on CAMO (Table 1), and AdaptFormer maintains a clear lead on SOD and HKU-IS (Table 6).

  2. Theoretical FLOPs do not equate to actual wall-clock inference acceleration, especially given that the reduction in FLOPs compared to DAR is merely 3.2%. Consequently, the paper's claim of "substantially reduced computation" lacks actual latency or throughput comparisons, making it an apparent overclaim.

  3. The 94.9% reduction in NMSE (from 22.32 to 1.13) to corroborate that their compensation mechanism is more effective. However, DAR's compensator was not designed or trained to approximate FFN residuals; benchmarking a baseline on an unoptimized surrogate metric is methodologically unfair. Furthermore, the paper lacks evidence showing a meaningful correlation between lower NMSE and downstream segmentation performance.

  4. The title targets general “pixel-level tasks”, but the validation is largely confined to binary segmentation tasks. The cross-task generalizability of TRACE across broader and more representative dense prediction benchmarks—such as multi-class semantic segmentation and depth estimation—remains unknown.

Questions for authors
  1. All results in Table 1 appear to come from a single run. I suggest the authors report the mean ± std over at least three random seeds for Ours vs. DAR, and indicate whether the reported 0.005–0.01 improvements are statistically significant?

  2. On CAMO, AdaptFormer w/o TR outperforms both Ours and Ours^SD, and it also leads on SOD and HKU-IS (Table 6). How do you explain this degeneration of TRACE?

  3. TRACE's FLOPs are only 3.2% lower than DAR's. Do you have wall-clock latency and peak memory measurements on the same GPU for AdaptFormer/DyT/DAR and TRACE?

  4. The title targets general "pixel-level tasks," yet all experiments are binary segmentation tasks. Do you have results on multi-class semantic segmentation or depth estimation? If not, would you consider narrowing the title/abstract claims to binary dense segmentation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HvhU

W1. Reliability and performance variation. We repeated DAR and Ours on DUTS-TE with three random seeds each (mean ± sample standard deviation):

MethodWeighted-FS-measureE-measure
DAR0.895553 ± 0.0006040.933750 ± 0.0002860.944100 ± 0.000339
Ours0.911757 ± 0.0006890.938139 ± 0.0006780.949130 ± 0.000572

The mean gains are 0.016204, 0.004390, and 0.005031, respectively, supporting the repeatability of the DUTS-TE result. To examine the broader trend, we computed the unweighted mean difference from AdaptFormer without TR across the six datasets in Table 1 (positive means higher than AdaptFormer):

DifferenceWeighted-FS-measureE-measure
DAR − AdaptFormer−0.0140−0.0037−0.0020
Ours − AdaptFormer+0.0020+0.0015+0.0015
Improvement over DAR+0.0160+0.0052+0.0035

Ours narrows the average performance gap associated with TR, although improvements are not uniform. On CAMO, Ours remains below AdaptFormer in Weighted-F, but the gap narrows from 0.024 (DAR) to 0.013 (Ours). The regressions on SOD and HKU-IS also remain. We therefore limit our claim to improvement across multiple datasets, rather than gains on every benchmark.

W2. Actual inference throughput. We measured all methods on an NVIDIA RTX 3090 using the same benchmarking procedure:

MethodFPSOurs vs. method
AdaptFormer36.31+5.98%
DyT36.78+4.62%
DAR37.12+3.66%
Ours38.48—

Ours reduces FLOPs by approximately 3.2% versus DAR and improves measured throughput by 3.66%. Versus AdaptFormer, measured throughput improves by 5.98%. We report actual throughput separately from theoretical FLOPs reduction.

W3. Interpretation of NMSE. We compare the compensation output with the missing FFN residual only for patch tokens that actually skip FFN computation. NMSE is a diagnostic of compensation behavior, not a reconstruction training objective or a segmentation metric. The complete per-layer data underlying Fig. 2(b) are:

LayerDAR skip rateDAR NMSEOurs skip rateOurs NMSE
L1<0.01%1.2649<0.01%1.0668
L2<0.01%1.18640%N/A
L3<0.01%1.59670%N/A
L431.36%1.21690.35%0.9114
L57.20%1.29508.05%0.9949
L633.50%1.236130.46%1.0137
L70.06%1.17690.01%1.0351
L829.72%1.267552.21%1.1292
L966.01%1.148769.61%1.1106
L1087.62%1.603781.57%1.1321
L11100%3.7539100%1.5402
L12100%22.3181100%1.1276

DAR's NMSE is about 1.1–1.6 in most layers but rises to 3.7539 and 22.3181 at L11/L12. Ours yields 1.5402 and 1.1276, respectively. Both methods skip 100% of patch-token FFN computation in these layers. Thus, Ours mitigates DAR's anomalous final-layer NMSE. DAR was not trained to fit FFN residuals, and Ours uses no dedicated FFN-residual reconstruction loss. Although Ours's L12 NMSE (1.1276) remains above the zero-compensation reference (~1.0), our analysis identifies and mitigates the anomalous NMSE spike observed in DAR. The relationship among NMSE, output scale, and segmentation performance warrants further investigation.

**W4. Beyond binary segmentation. ** We additionally evaluated multi-class semantic segmentation on ADE20K:

MethodBest mIoU
DAR45.92%
Ours46.31%

The 0.39-percentage-point gain provides initial evidence beyond binary segmentation.We will further evaluate the method on depth estimation in future work.

Q1. Are the gains reliable? See W1: both methods were run with three seeds on DUTS-TE, and Ours has higher mean scores on all three metrics. We do not extrapolate this repeatability result to every dataset.

Q2. What explains the regressions? See W1. CAMO's Weighted-F gap to AdaptFormer narrows from 0.024 to 0.013, but regressions on SOD and HKU-IS remain. Our conclusion concerns the overall trend across datasets, not a gain on every benchmark.

Q3. Is there actual speedup? See W2: Ours reaches 38.48 FPS versus DAR's 37.12 FPS on the RTX 3090, a 3.66% throughput gain.

Q4. Does the method work beyond binary tasks? See W4: Ours achieves 46.31% best mIoU on ADE20K versus DAR's 45.92%. This is preliminary multi-class evidence; depth estimation remains untested.

Thanks for the authors' feedback. The ADE20K result and the throughput measurement address parts of my concerns. However, my two main concerns remain.

First, the three-seed experiment is limited to DUTS-TE, and the authors appropriately state that they do not extrapolate this repeatability result to every dataset. The response on CAMO, SOD, and HKU-IS acknowledges the remaining regressions but does not explain them.

Second, the NMSE clarification addresses only part of my concern, No evidence is provided that lower NMSE correlates with downstream segmentation performance.

Finally, the 3.66% throughput gain over DAR is useful but modest, so “substantially reduced computation” remains an overstatement in this comparison.

Overall, I maintain my score.

Reviewer_y6EZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work addresses the limitations of token reduction in parameter-efficient fine-tuning.

The authors' analysis shows a design gap under token reduction.

To address this gap, the propose a framework that uses the tokens via token reduction to regulate compensation and adaptation for improved PEFT.

They show improved performance compared to baselines.

Critical strengths
  • This paper is well motivated.

  • The evaluations show improved performance compared to baselines.

  • The paper presents some interesting visualisations eg fig 4.

Critical weaknesses

Major:

  • The term compensation under PEFT is unclear to me. The authors should explain explicitly what is the meaning of compensation under PEFT in intro.

  • Fig 1 does not clearly show the novelty of this work, and what are the key insights of the proposed framework. It seems more like an integration of existing frameworks.

  • Intro has some redundancy and reads like machine generated due to the summary at the end of page 2. The claimed contributions sound weak, particularly the first two points. It is unclear how these two mechanisms such as compensation and adaptation advance PEFT compared to existing works in the literature.

  • Related work section is quite short, and poorly written. The authors simply listed existing works per group, without a detailed analysis of (i) what are the remaining challenges or weaknesses of these methods, and (ii) how the proposed method differs from existing methods.

  • Fig 3 as the main pipeline overview could be much clearer and cleaner. The current visual presentation does not clearly show the innovations. In addition, this work introduces too many hyperparameters, for example Eq. 14 and 15, those loss terms require careful tuning of the weighting parameters, making the framework a bit impractical.

  • The performance improvements appear to be marginal, eg, in table 1. Also the evaluations presented in the current paper appear to be very limited. The datasets used for evaluation also appear to be too small scale.

  • Most experimental results are presented in the form of tables, and more visualisations that analyse why the proposed mechanisms are needed.

Minor:

  • Fig font sizes appear to be too small to read, eg, fig 1 and 2.

  • Table font sizes could be consistent in the whole paper.

Questions for authors

Refer to above raised concerns under major categories.

2 comments
AuthorsOpenReview ↗

Response to Reviewer y6EZ

Response to Reviewer y6EZ

Thank you for the review. We clarify the motivation, design, and experimental evidence below.

W1. Meaning of compensation. To accelerate inference, TR allows some tokens to skip FFN computation in the frozen backbone. Compensation aims to mitigate the resulting information loss. In TRACE, the compensation branch provides an additional feature residual for skipped tokens, while the PEFT branch performs downstream task adaptation. The introduction discusses the skipped FFN computation and information loss; this distinction between the two branches clarifies what “compensation” means in our setting.

W2. Figure 1 and novelty. Figure 1 contrasts the design problem with our solution. Existing TR-PEFT methods create preserved and skipped token states, but do not explicitly use these states to coordinate compensation and adaptation. TRACE uses the states to apply selective compensation to skipped tokens and coordinate their compensation and PEFT adaptation responses. The contribution is this state-aware organization of the two pathways.

W3. Introduction and contributions. The recap at the end of the introduction connects the observed behavior of existing methods to TRACE's design. Specifically, Fig. 2 shows an anomalous compensation discrepancy in deeper layers and a weakened boundary/non-boundary response difference after TR. These observations motivate using TR-induced token states to regulate compensation and adaptation, as summarized in the contributions.

W4. Related work. The related work section summarizes the existing approaches, while the introduction analyzes the limitation relevant to our method. Methods that strengthen adaptation or add a compensation pathway address information loss under TR. Our analysis focuses on a further design issue: these pathways do not explicitly use the preserved/skipped token states to coordinate compensation and adaptation. TRACE addresses this issue through selective compensation and adaptive mixing.

W5. Figure 3 and training complexity. In Fig. 3, preserved tokens retain the PEFT pathway, whereas skipped tokens receive selective compensation coordinated with PEFT adaptation. The equations specify how these paths are combined. Although Eqs. (14)–(15) display several loss weights, most hyperparameters use fixed settings across our experiments; the method does not depend on extensive tuning of these weights.

W6. Performance gains. We follow the same or comparable segmentation benchmarks used in prior TR-PEFT work to enable direct comparison. In Table 1, Ours improves Weighted-F over DAR on all six datasets, by 0.016 on average, while reducing FLOPs from 130.01 G to 125.82 G. These results support a performance–efficiency improvement on the evaluated benchmarks; we do not extend the claim to untested tasks or datasets.

W7. Visual evidence. Fig. 2 provides quantitative analyses of compensation and adaptation responses. The existing appendix visualizations (Figs. 5–11) further show token-selection behavior and segmentation outputs across tasks. Together, they complement the tabulated results, although they do not directly visualize every internal mechanism.

Minor presentation issues. We acknowledge the concern that some text in Figs. 1 and 2 is small and that table font sizes vary.

I have read the author’s responses, as well as the review comments from other reviewers.

Considering all, I think my current rating fairly reflects the current version of the paper, I maintain my original rating.

Reviewer_pkzQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a state-aware token-reduction framework for PEFT in pixel-level segmentation. By introducing Selective Compensation (SCM), Shared Down-Projection (SDP), and Adaptive Mixing (AMM), the system aims to recover information lost through FFN skipping while improving computational efficiency. Experiments across salient-object, camouflaged-object, and medical segmentation show promising accuracy–FLOP trade-offs.

Critical strengths

The proposed state-aware compensation and adaptation framework is straightforward and technically coherent, explicitly differentiating preserved and skipped tokens.

The method is evaluated across salient-object, camouflaged-object, and medical segmentation datasets, providing evidence of applicability across different binary segmentation domains.

The paper presents a well-structured architecture and easy to follow.

Critical weaknesses

The novelty is incremental relative to DAR. The Token Selector, the post-FFN mask during training, the decoder, and the decoupled compensation idea are all inherited from DAR

The authors argue that token reduction weakens the difference in adaptation strengths between boundary and non-boundary tokens, but do not demonstrate that TRACE restores this distinction. Furthermore, differences in residual magnitude do not necessarily indicate better task adaptation.

The ablations compare adaptive mixing only against fixed 0.5/0.5 weights, without evaluating learned global or layer-wise mixing. Consequently, the improvements cannot be conclusively attributed to token-dependent state awareness rather than more flexible residual weighting.

The effectiveness of FFN residual reconstruction is still questionable. The NMSE comparison may not be apples-to-apples and and its relevance is still unclear. TRACE also has no loss that targets the FFN residual. It is unclear why its compensation should approximate it. The reported NMSE improvement from 22.32 to 1.13 is insufficient to demonstrate effective compensation since it could partly reflect scale differences between modules rather than better “approximation.”

Questions for authors

How do you justify effective FFN reconstruction when TRACE's NMSE of 1.13 is worse than the approximately 1.0 achieved by zero compensation?

Does TRACE restore the boundary/non-boundary adaptation-strength differences identified in the motivation, and can this be quantitatively demonstrated?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pkzQ

Response to Reviewer pkzQ

W1. Novelty. TRACE inherits DAR's Token Selector, training-time post-FFN mask, decoder, and decoupled compensation framework. DAR primarily motivates its compensation design by the information loss introduced by TR; our insight is that TR-induced token states should guide the coordination of compensation and adaptation. We therefore propose a state-aware TR-PEFT architecture.

W2. Boundary/non-boundary responses. We apologize that the original paper did not directly show TRACE's effect on their adaptation-strength difference. We used the same data, setup, boundary definition, and RRS calculation as Fig. 2(c)–(d):

LayerAdaptFormer+TR boundaryAdaptFormer+TR non-boundaryTRACE boundaryTRACE non-boundary
L1-16.3%+19.4%-13.3%+15.3%
L2-0.8%+0.8%-1.6%+1.6%
L3-8.2%+9.0%-1.6%+1.6%
L4+1.8%-1.8%-1.8%+1.8%
L5+6.1%-5.7%+7.2%-6.8%
L6+0.6%-0.6%-5.5%+5.9%
L7-1.4%+1.5%+4.9%-4.7%
L8-0.6%+0.6%+1.1%-1.1%
L9-1.0%+1.0%+2.1%-2.1%
L10-0.5%+0.5%+2.4%-2.3%
L11-0.5%+0.5%-9.8%+10.8%
L12+0.2%-0.2%-1.1%+1.1%

In L4–L12, where skipping occurs, TRACE has a larger boundary/non-boundary response difference in 8 of 9 layers. The mean absolute gap between the two RRS deviations rises from 2.79 to 8.06 percentage points; 7 layers match the direction in vanilla AdaptFormer (Fig. 2(c)). Thus, differentiated responses are partially restored. We use vanilla AdaptFormer, a strong PEFT baseline without TR, as the behavioral reference: this difference weakens after TR is introduced. This motivated TRACE's design. The RRS analysis shows partial recovery of the response pattern, and segmentation metrics improve over existing TR-PEFT methods; residual magnitude alone does not establish better task adaptation.

W3. Mixing without state guidance. We replaced only TRACE's AMM with adaptive weights generated per token for mixing the PEFT and compensation branches, without guidance from each token's preserved/skipped state. Other settings remain unchanged. The control was run with three seeds:

MethodWeighted-FS-measureE-measure
DAR0.8950.9330.944
State-agnostic token-wise mixing (3 seeds)0.892439 ± 0.0006700.933043 ± 0.0001020.944029 ± 0.000594
TRACE0.9120.9380.949

The state-agnostic control is comparable to DAR and below TRACE on all three metrics. Thus, adaptive weights alone do not explain the gain; token-state-guided coordination contributes to it.

W4. NMSE and missing FFN residuals. TR loses information when tokens skip FFN computation. We measure NMSE between the compensation output and missing FFN residual on patch tokens that actually skip FFN. This diagnoses compensation behavior; it is not TRACE's training objective. The full per-layer data underlying Fig. 2(b) are:

LayerDAR skip rateDAR NMSETRACE skip rateTRACE NMSE
L1<0.01%1.2649<0.01%1.0668
L2<0.01%1.18640%N/A
L3<0.01%1.59670%N/A
L431.36%1.21690.35%0.9114
L57.20%1.29508.05%0.9949
L633.50%1.236130.46%1.0137
L70.06%1.17690.01%1.0351
L829.72%1.267552.21%1.1292
L966.01%1.148769.61%1.1106
L1087.62%1.603781.57%1.1321
L11100%3.7539100%1.5402
L12100%22.3181100%1.1276

DAR's NMSE is about 1.1–1.6 in most layers but rises to 3.7539/22.3181 at L11/L12. TRACE reduces these to 1.5402/1.1276. Both methods skip all patch-token FFN computations there. Relative to DAR, TRACE gives a more stable approximation of the missing FFN residual in these high-skipping layers, mitigating the anomalous error. TRACE adds no dedicated FFN-reconstruction module or loss. NMSE diagnoses this change; segmentation results assess task performance.

Q1. NMSE 1.13 versus ~1.0 for zero compensation? As discussed in W4, an NMSE of 1.13 does not demonstrate accurate FFN-residual reconstruction relative to zero compensation. Relative to DAR, however, TRACE provides a more stable approximation of the missing FFN residual in high-skipping layers, eliminating DAR's anomalous L12 spike in NMSE (22.32 to 1.13) while also improving segmentation performance.

Q2. Restored boundary/non-boundary differences? As in W2, 8 of 9 skipping layers show a larger gap (mean 2.79 to 8.06 percentage points), and 7 match vanilla AdaptFormer's direction. This supports partial recovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.