acceptodds
Under review as a conference paper at ICLR 2027

HazeMT-DCR: Demand-Aware Counterfactual Expert Routing for Incremental Multi-Task Prediction in Hazy Scenes

Abstract

Existing multi-task perception methods for hazy scenes usually adopt a two-stage “dehazing-then-prediction" paradigm. However, dehazing objectives for visual restoration or pixel reconstruction may not match downstream task requirements. A unified shared representation may also fail to capture the different semantic, structural, and geometric cues required by different tasks. To address these issues, we propose HazeMT-DCR, a demand-aware counterfactual expert routing framework for multi-task prediction in hazy scenes. We reformulate hazy multi-task perception as shared functional reuse and dynamic expert composition conditioned on task demands. Without explicit clear-image restoration, HazeMT-DCR calibrates multi-scale visual features using latent degradation states and conditions shared functional experts on task demands, degradation states, and image content. Each task learns an independent reinforcement learning routing policy to dynamically combine the shared experts. To improve routing reliability, we introduce a task-demand intervention counterfactual regret mechanism. Expert removal identifies redundant or negatively contributing experts, while task-demand replacement checks whether expert responses depend on the correct task demand. The resulting counterfactual signals are used to improve expert routing. HazeMT-DCR also decouples shared modules from task-specific parameters, allowing new tasks to be adapted by training only task-specific routing and output modules without retraining the full model. We evaluate HazeMT-DCR on four dense prediction tasks on hazy PASCAL-Context. Experiments show that HazeMT-DCR outperforms existing parameter-efficient multi-task adaptation methods while requiring only a small number of trainable parameters, and supports efficient task expansion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-task prediction, incremental task adaptation, task-demand-driven reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1DbJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HazeMT-DCR addresses multi-task prediction in hazy scenes without explicit image restoration. It combines degradation-aware feature modulation, instruction-conditioned shared experts, and task-specific reinforcement-learning routing. Expert-removal and task-demand-replacement counterfactual losses further guide training. Experiments on synthetically hazed PASCAL-Context report improved overall performance with 5.08M trainable parameters. Incremental adaptation experiments, including an additional detection task on real hazy RTTS images, evaluate reuse of frozen shared modules.

Critical strengths
  1. The framework directly addresses task-dependent feature requirements under degradation. Counterfactual expert removal and instruction replacement provide concrete training signals beyond the final prediction loss, and the reported ablations support their usefulness.
  2. The evaluation covers four dense prediction tasks, multiple task-expansion configurations, and real-haze object detection. The reported performance–parameter tradeoff is promising, and the real-data extension strengthens the practical relevance.
Critical weaknesses
  1. The incremental evaluation needs clarification. The paper states that all shared modules and existing task paths remain frozen, yet old-task results change after expansion (Tables 2 and 4). With identical inputs, instructions, model states, and deterministic inference, those predictions should remain unchanged. Changes in instruction composition or evaluation conditions could explain this, but must be documented to interpret the retention results.
  2. The benefits of the optimization components are not fully isolated. The actor receives both PPO and direct reparameterized gradients, while the counterfactual losses also update the expert network. Although a Softmax-routing comparison is provided, it remains unclear whether PPO adds value over the same stochastic router trained only through differentiable losses. Moreover, if gradients flow through counterfactual losses, regret could decrease by worsening counterfactual predictions rather than improving the factual prediction.
Questions for authors
  1. What causes old-task predictions to change when their entire computation paths are frozen? Are instructions, normalization statistics, and evaluation inputs identical before and after expansion?
  2. Does PPO improve over reparameterized Dirichlet routing trained with the same task and counterfactual objectives but without PPO? Please include training time and repeated-run variability.
  3. Are the counterfactual comparison losses detached when optimizing LcfL_{\mathrm{cf}}? Can you show that reduced regret corresponds to better factual predictions rather than degraded counterfactual branches?
3 comments

We thank the reviewer for the careful assessment and for highlighting the need to clarify incremental evaluation and better isolate the effects of PPO and counterfactual optimization. Below, we address, in order, old-task prediction consistency, the contribution of PPO beyond differentiable Dirichlet routing, and whether reduced counterfactual regret is accompanied by improved factual predictions.

  1. What causes old-task predictions to change when their entire computation paths are frozen? Are instructions, normalization statistics, and evaluation inputs identical before and after expansion?(Weakness 1 ,Question 1)

We need to clarify that, for our method, the old-task evaluation protocols before and after expansion are not exactly the same. The two settings use the same evaluation state but different task instructions and different checkpoints, one before expansion and one after expansion; the parameters of the shared modules and existing task modules remain identical. Specifically: Old-task Only stage: We train the model using only old-task data and evaluate its predictive performance using the combined instruction for the old tasks. Incremental Adaptation stage: Starting from the trained old-task model, we freeze HASE, TCM, ICFE, and the routing networks and output heads of all existing tasks, introduce and train only the corresponding task-specific modules for the newly added tasks, and evaluate its predictive performance using the expanded four-task combined instruction. Therefore, the old-task evaluation protocols before and after expansion are not exactly the same.

Regarding normalization statistics, Our model uses no BatchNorm: LayerNorm is applied in the Swin backbone and expert control vectors, GroupNorm in the degradation estimator, FPN, decoder, and prediction heads, and no normalization in the PPO router. Thus, no running statistics require synchronization. Validation uses fixed 448×448 inputs with ImageNet normalization and no random augmentation. All evaluations use eval(), torch.no_grad(), and deterministic Dirichlet-mean routing via act_deterministic(), without sampling.

We emphasize that the purpose of using different task combination instructions before and after expansion is to measure end-to-end performance under actual task expansion conditions. Before expansion, the model only needs to perform the original tasks, so it is evaluated using the combined instruction for the old tasks. After expansion, however, the model should be capable of performing all four tasks simultaneously. We therefore evaluate the expanded model using a joint four-task instruction and examine how well it retains its performance on the old tasks while performing all tasks. Compared with consistently using the old-task instruction, this evaluation better reflects the model's performance under full multitask execution conditions after new tasks are added.

To distinguish prediction preservation under a fixed instruction from performance changes under actual joint four-task execution conditions, we have followed the reviewer's suggestion and added the Old-Instruction Evaluation setting to Table 4 in the revised manuscript. We have reevaluated our model using the following three settings:

Evaluation SettingCheckpointTask Instruction
Old-task OnlyPre-expansion checkpointOld-task instruction
Incremental AdaptationExpanded checkpointFour-task instruction
Old-Instruction EvaluationExpanded checkpointOld-task instruction

The experimental results in Table 4 of the revised manuscript show that, across all ten task expansion settings, the old-task performance is exactly the same under Old-task Only and Old-Instruction Evaluation. Moreover, with the same input images and task instructions and under deterministic inference conditions, the maximum absolute difference in the raw prediction outputs is 0, further verifying the consistency of old-task predictions under fixed evaluation conditions.

In contrast, Incremental Adaptation uses the four-task combined instruction, so old-task performance may change. These control results show that, across the ten task expansion settings tested, both old-task performance and raw prediction outputs remain identical when the same old-task instruction is used before and after expansion. However, for the same expanded checkpoint, switching from the old-task instruction to the four-task combined instruction may change old-task performance. Therefore, the performance changes in the original table can be explained by the change in the task combination instruction.

We have further clarified the relevant evaluation protocols in Section 4.3 and Appendix A.4 of the revised manuscript and added the fixed-instruction control results to Table 4 to resolve any ambiguity in the original text.

  1. Does PPO improve over reparameterized Dirichlet routing trained with the same task and counterfactual objectives but without PPO? Please include training time and repeated-run variability.(Weakness 2 ,Question 2)

We agree that the original Softmax routing comparison changes both the routing parameterization and the optimization method, and is therefore insufficient to isolate the contribution of PPO. To address this, we have added a Dirichlet Routing w/o PPO control to the revised manuscript and report the mean performance, standard deviations, and average training time across three runs in Table 7. The results show that, while retaining Dirichlet reparameterized sampling and the direct gradients from the task loss and counterfactual loss, adding PPO updates further improves the mean performance of all four tasks, with a 7.49% increase in the time cost of the full training procedure. Details are provided below:

The newly added control retains the same Dirichlet routing formulation as the full model. Both configurations use reparameterized sampling and retain the gradients from the task loss Ltask\mathcal{L}_{\mathrm{task}} and counterfactual loss that propagate to the actor through the expert weights wt\mathbf{w}_t. The control without PPO still includes stochastic routing exploration and can directly learn the routing policy through differentiable losses; it also retains the original counterfactual supervision. To control for the effects of initialization and training randomness, the two configurations use matched warm-up checkpoints and random seeds. Apart from removing PPO policy updates, the routing parameterization, expert and prediction modules, task and counterfactual supervision, and established training settings remain the same. The newly added control disables PPO policy updates while retaining the use of the aforementioned differentiable objectives to train the actor and prediction network. Both configurations use the same number of training epochs: a total of 400 epochs, consisting of 200 epochs of expert warm-up and 200 epochs of joint optimization, with a batch size of 16. The experiments are conducted on a single NVIDIA GeForce RTX 4090 (24 GB). Therefore, the performance comparison is based on the same training epoch budget, while differences in actual elapsed time are reported separately.

As shown in Table 7, the full model improves the mean performance on every task compared with reparameterized Dirichlet routing without PPO.

Because the newly added control already uses the same Dirichlet reparameterized sampling and receives direct supervision from the task and counterfactual losses, these results support the additional empirical benefits of PPO beyond direct differentiable optimization. This conclusion no longer depends on the original Softmax routing comparison.

Regarding variability across repeated runs, we also note that the full model has higher standard deviations than the control without PPO on all four metrics. Thus, this experiment supports improvements in mean task performance, but it does not justify a claim that PPO reduces variance across runs.

We have clarified reparameterized sampling and PPO gradient handling in Sections 3.4–3.5. Appendix A.2 explains that the actor receives both direct differentiable gradients and PPO gradients; Appendix A.3 provides the training settings; and the "Isolating the Contribution of PPO" paragraph in Appendix A.5 and Table 7 present the newly added control, variability across three runs, and training time. We thank the reviewer for this suggestion, which has enabled more direct and transparent verification of PPO's incremental contribution and training cost.

  1. Are the counterfactual comparison losses detached when optimizing LcfL_{\mathrm{cf}}? Can you show that reduced regret corresponds to better factual predictions rather than degraded counterfactual branches?(Weakness 2 ,Question 3)

To first answer the question about gradient settings directly: when optimizing Lcf\mathcal{L}_{cf}, the original method does not detach either the expert removal loss Lt(−j)\mathcal{L}_t^{(-j)} or the task demand replacement loss Ltc′→c\mathcal{L}_t^{c'\rightarrow c}. The Counterfactual Regret Loss remains differentiable, allowing the expert modules to be updated and gradients to propagate to the actor network through the reparameterized expert weights.

We understand the reviewer's concern: the identification and correction mechanism described above does not imply that every decrease in the hinge objective necessarily results from improvements in factual predictions. When gradients are retained for the counterfactual losses, it is indeed necessary to examine whether regret is reduced by increasing the counterfactual losses. We have therefore added the following analyses that directly examine the optimization behavior.

In the Counterfactual Optimization Analysis in Appendix A.5, we use the same batch of 64 validation images to evaluate the checkpoints at epochs 201 and 400, reporting both types of counterfactual regret, the three types of branch losses, and task performance.

The results in Table 10 show that both types of regret decrease for all four tasks, while task performance based on factual predictions improves for all four tasks. In addition, the factual loss and both types of counterfactual losses decrease for all tasks, and within each task, the absolute decrease in the factual loss is greater than the absolute decrease in either type of counterfactual loss. In particular, the loss of the task demand replacement branch, which the reviewer specifically highlighted, also decreases for all four tasks, with no overall increase.

These results show that, for the reported fixed validation subset and the two checkpoints evaluated, reductions in regret are accompanied by lower factual prediction losses and improved task performance, while both types of counterfactual predictions also improve. Thus, the observed results do not support the explanation that "regret is reduced solely by degrading counterfactual predictions overall," and provide direct empirical support for the effectiveness of the expert learning and combination correction mechanism.

Following the reviewer's suggestion, Table 6 in the revised manuscript compares the original full model with the TICR w/ Detached CF Losses variant. In the latter, gradients are stopped for both types of counterfactual prediction losses when constructing Lcf\mathcal{L}_{cf}, while gradients for the factual prediction loss are retained.

The results in Table 6 show that the original model performs better on all four tasks. This indicates that, under the experimental settings in this paper, retaining counterfactual gradients helps achieve better final task prediction performance.

We need to clarify that what we refer to as a "harmful contribution" applies only to the current sample, task, and weight configuration: if the loss decreases after an expert is removed and the remaining weights are renormalized, this indicates that the current combination has room for improvement; it does not mean that the expert is harmful to all samples and tasks. During training, the task loss and the differentiable counterfactual loss jointly optimize the experts and the actor, while PPO further optimizes the routing policy. Thus, this mechanism adjusts both expert responses and how they are combined to improve factual predictions, rather than merely reducing the probability of a particular combination, and it does not guarantee that every update will reduce the weight of a particular expert.

Expert removal provides an explicit criterion for identifying harmful contributions within the current combination, while the regret penalty and PPO feedback provide an optimization mechanism for correcting expert combinations. The newly added Tables 10 and 6 provide empirical support through an analysis of optimization behavior and a comparison of gradient settings, respectively. We do not extrapolate these results into a guarantee of monotonic improvement for every update or every sample. However, they support the view that TICR uses counterfactual diagnosis to improve expert combinations and factual predictions, rather than relying on overall degradation of the counterfactual branches.

Reviewer_SYAEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for direct multi-task prediction and incremental task expansion on hazy images. The model conditions shared features on latent degradation states, controls experts using textual task requirements and image content, and combines expert outputs via task-specific routing. During training, it introduces counterfactual regret constraints produced by expert removal and task-requirement substitution, together with PPO routing optimization. When adding a new task, the shared modules and old-task parameters are frozen, and only the new task modules are trained. Experiments include synthetic hazy PASCAL-Context, several task-expansion settings, and real-world haze object detection.

Critical strengths

The paper combines direct task prediction with incremental task expansion, which has practical significance. The separation between shared experts and task-specific modules offers a concrete implementation for reusing existing representations.

The paper not only reports the full system's performance, but also provides ablations of several components and internal mechanisms, including a Softmax dynamic-routing comparison, expert-modulation decomposition, and analysis of the two counterfactual regret terms, and additionally includes a real-world haze detection extension experiment.

Critical weaknesses

Removing the degradation branch degrades performance, indicating that this branch provides useful information; however, it does not necessarily prove that the information primarily corresponds to haze degradation rather than general image content. The effectiveness of the text condition also does not necessarily show that language semantics are superior to task identity. PPO outperforms the reported Softmax dynamic routing, but it still needs to be clarified whether the two use the same counterfactual supervision, training schedule, and matched routing capacity.

The authors explicitly state that the shared modules and old-task parameters are frozen when adding a new task. If the images, instructions, running state, and deterministic inference process for old tasks all remain unchanged, their outputs should in principle remain identical. However, the old-task metrics still change in Table 2, e.g., semantic segmentation drops from 70.52 to 70.11 and saliency detection from 63.76 to 63.43. Do the combined instructions received by old tasks change before and after a new task is added? If they do, the metric drop may stem from changes in the conditioning input, rather than solely from forgetting caused by learning the new task.

Questions for authors
  1. Does the comparison between Softmax dynamic routing and PPO routing change only the routing optimization mechanism? The existing experiments are not sufficient to determine which specific mechanisms constitute the main contribution.
  2. Does the text condition provide any benefit beyond task identity? The current task set is limited and uses predefined instructions. Have the authors compared replacing the text encoding with a task index or a learnable task embedding, while keeping the subsequent conditioning modules and training protocol unchanged?
  3. The experiments removing the branch and altering the conditioning input support its effectiveness, but is there evidence that this vector is sensitive to haze degradation rather than primarily encoding scene category or general visual content? For example, an analysis of the responses for the same scene under different haze intensities, or an examination of the degradation vector and routing changes while controlling for scene content.
5 comments

We thank the reviewer for identifying aspects that require stronger control experiments and clearer interpretation of the results. In response to these comments, we have added ablation experiments and further clarified the evaluation protocols in the revised manuscript. Below, we address, in order, the routing comparison, task conditioning representations, the fog sensitivity of the degradation vector, and the consistency of old-task predictions under frozen paths.

  1. Comparison of PPO and Softmax Routing(Weakness 1 ,Question 1)

We agree that comparing Softmax routing with PPO routing alone is insufficient to isolate the contribution of PPO, because this comparison changes both the parameterization of the routing weights and the policy optimization method.

To clarify the control conditions of the Softmax comparison, we specify in Appendix A.5 that Softmax dynamic routing and the full model use the same warm-up initialization, task and counterfactual supervision, training schedule, matched actor architectures, and four-expert library. Both configurations undergo 200 epochs of warm-up followed by 200 epochs of routing optimization. Importantly, the Softmax baseline also receives counterfactual supervision rather than being optimized solely with the task loss.

Table 7 in the revised manuscript further compares the full model with a Dirichlet routing variant without PPO updates. The two configurations use matched warm-up checkpoints and random seeds, and both retain Dirichlet reparameterized sampling and direct gradients to the actor from the task loss and counterfactual loss. The means and standard deviations from three independent runs are also reported.

As shown in Table 7 of the revised manuscript, adding PPO improves mIoU by 0.90, 0.84, and 0.37 percentage points for semantic segmentation, human part segmentation, and saliency detection, respectively, and reduces mErr by 0.69 for surface normal estimation. The average training time, including the warm-up stage, increases from 21.62 hours to 23.24 hours, corresponding to a time overhead of 7.49%.

The contribution of counterfactual supervision is examined through a separate set of ablation experiments: removing TICR reduces Δm\Delta m from 3.96%3.96\% to 2.36%2.36\%, and Figure 5 further analyzes its internal components. Thus, the existing ablation results support contributions from both counterfactual constraints and routing optimization; we do not attribute all gains of the full system solely to PPO.

  1. Whether Text Conditioning Outperforms the Tested Task Identity Encoding(Weakness 1 ,Question 2)

We agree that the experiment removing the text branch only shows that conditioning information plays a role; it cannot establish that language semantics are superior to task identity information. To address this, we have added a direct comparison with a task identity baseline while keeping the architecture of the downstream conditioning modules and the training protocol unchanged. The results are presented in Appendix A.5 and Table 8 of the revised manuscript.

The four tasks form a total of 15 nonempty task combinations. BERT text conditioning uses a frozen BERT encoder to encode the instruction corresponding to each task combination into a 768-dimensional text embedding, which is then mapped to a 128-dimensional task representation through a two-layer adapter. Multi-Hot conditioning explicitly represents the set of active tasks using a four-dimensional binary vector, projects it to 768 dimensions through a linear layer without an activation function, and then uses an adapter with the same architecture. Thus, this baseline explicitly preserves task identity and task combination information. The two configurations are trained separately under the same settings, and their reported numbers of trainable parameters are similar, both at 5.08M.

As shown in Table 8 of the revised manuscript, BERT conditioning improves the metrics for all four tasks compared with this baseline that explicitly provides task identity. This indicates that pretrained text representations can still provide performance gains when task identity information remains available and the downstream conditioning modules remain unchanged.

At the same time, we have more clearly qualified our interpretation of this result. This experiment supports the empirical benefits of pretrained text representations over the tested multi-hot encoding. It does not yet distinguish whether these benefits arise from language semantics, the pretrained representations themselves, or the associated inductive biases, nor does it establish that BERT outperforms all possible learnable task embeddings. The comparison of 5.08M parameters above concerns only trainable parameters; BERT itself remains frozen. Appendix A.5 of the revised manuscript explicitly clarifies the scope of this conclusion and does not present it as independently established evidence of an advantage from language semantics.

  1. Whether the Degradation Vector Is Indeed Sensitive to Fog Degradation(Weakness 1 ,Question 3)

We agree that an ablation experiment removing a branch cannot directly establish what information that branch encodes.

We need to clarify that we have neither defined hh as a feature sensitive to haze nor used images of foggy scenes to constrain hh. hh is simply a latent descriptor vector that is sensitive to fog variations in foggy scenes and is used for feature adaptation. hh can contain both haze information and general scene information; we constrain this variable only through the final model performance.

Following the reviewer's suggestion, we have added a controlled analysis in which scene content is held fixed and only fog intensity is varied. Details are provided in Appendices A.3 and A.5 and Table 9 of the revised manuscript.

For 256 validation scenes, we generate images with light, moderate, and heavy fog by setting only the scattering coefficient β\beta to 0.07824, 0.1956, and 0.7824, respectively, while holding scene content, depth, the noise field, and atmospheric light fixed. We use the model checkpoint at epoch 400 and evaluate it with the same task instruction and deterministic Dirichlet mean routing, measuring the cosine distance for the latent descriptor vector hh and the L1L_1 distance for each task's routing weight vector. The table below lists the mean values from Table 9 in the revised manuscript; the original table also reports standard deviations.

As shown in Table 9 of the revised manuscript, for the same scene, the mean distance between degradation representations increases markedly when the change in fog intensity involves the heavy-fog setting. For all four tasks, the mean routing difference between "light fog and heavy fog" is also greater than that between "light fog and moderate fog." Because scene content is held fixed, these results directly support the sensitivity of the degradation representation and routing to changes in fog intensity.

We also construct 128 disjoint pairs of images from different scenes and compare them at each of the three matched fog levels, yielding 384 comparisons in total. These results also show that the model remains influenced by scene content. For example, at the same fog level, the mean distance between hh vectors from different scenes is 0.510133; the routing difference for semantic segmentation across scenes is even greater than that between light fog and heavy fog within the same scene.

Together with the ablation results for the degradation branch, this evidence supports using hh as a latent descriptor vector that is sensitive to fog variations and is used for feature adaptation. Section 3.2 of the revised manuscript defines it as a data-driven latent degradation descriptor, rather than a physically calibrated estimate of fog density or atmospheric light. Appendix A.5 also explicitly limits the experimental conclusion to fog sensitivity and does not claim that this representation has been disentangled from scene content.

  1. Changes in Old-Task Metrics under Frozen Prediction Paths(Weakness 2 )

We need to clarify that, for our method, the old-task evaluation protocols before and after expansion are not exactly the same. The two settings use the same evaluation state but different task instructions and different checkpoints, one before expansion and one after expansion; the parameters of the shared modules and existing task modules remain identical. Specifically: Old-task Only stage: We train the model using only old-task data and evaluate its predictive performance using the combined instruction for the old tasks. Incremental Adaptation stage: Starting from the trained old-task model, we freeze HASE, TCM, ICFE, and the routing networks and output heads of all existing tasks, introduce and train only the corresponding task-specific modules for the newly added tasks, and evaluate its predictive performance using the expanded four-task combined instruction. Therefore, the old-task evaluation protocols before and after expansion are not exactly the same.

We emphasize that the purpose of using different task combination instructions before and after expansion is to measure end-to-end performance under actual task expansion conditions. Before expansion, the model only needs to perform the original tasks, so it is evaluated using the combined instruction for the old tasks. After expansion, however, the model should be capable of performing all four tasks simultaneously. We therefore evaluate the expanded model using a joint four-task instruction and examine how well it retains its performance on the old tasks while performing all tasks. Compared with consistently using the old-task instruction, this evaluation better reflects the model's performance under full multitask execution conditions after new tasks are added.

To distinguish prediction preservation under a fixed instruction from performance changes under actual joint four-task execution conditions, we have followed the reviewer's suggestion and added the Old-Instruction Evaluation setting to Table 4 in the revised manuscript. We have reevaluated our model using the following three settings:

Evaluation SettingCheckpointTask Instruction
Old-task OnlyPre-expansion checkpointOld-task instruction
Incremental AdaptationExpanded checkpointFour-task instruction
Old-Instruction EvaluationExpanded checkpointOld-task instruction

The experimental results in Table 4 of the revised manuscript show that, across all ten task expansion settings, the old-task performance is exactly the same under Old-task Only and Old-Instruction Evaluation. Moreover, with the same input images and task instructions and under deterministic inference conditions, the maximum absolute difference in the raw prediction outputs is 0, further verifying the consistency of old-task predictions under fixed evaluation conditions.

In contrast, Incremental Adaptation uses the four-task combined instruction, so old-task performance may change. For example, when SemSeg and Saliency are the old tasks, their mIoU values remain at 70.52 and 63.76, respectively, under the fixed instruction, whereas under the joint four-task instruction, they are 70.11 and 63.43, respectively. This indicates that the changes in old-task performance observed in the original experiments are associated with changes in the task combination evaluation conditions, rather than updates to existing task parameters during incremental training.

We have further clarified the relevant evaluation protocols in Section 4.3 and Appendix A.4 of the revised manuscript and added the fixed-instruction control results to Table 4 to resolve any ambiguity in the original text.

Reviewer_tRBiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HazeMT-DCR performs dense prediction directly from hazy images using degradation-conditioned features, instruction-modulated experts, and task-specific Dirichlet routing policies. Expert-removal and instruction-replacement losses regularize routing, while incremental adaptation freezes shared and existing task modules and trains new task modules.

Critical strengths

The method reports improved aggregate performance with modest trainable-parameter overhead, and the appendix includes component ablations and an additional real-haze detection task. These evaluations are relevant to task adaptation beyond image dehazing.

Critical weaknesses
  1. The incremental results are not reconciled with the claimed frozen prediction paths (Appendix A.4; Tables 2 and 4). If the shared modules, old-task parameters, inputs, and instructions are unchanged, deterministic old-task outputs should be unchanged as well. Yet Table 4 reports, for example, Parts dropping from 59.28 to 58.26 and Normal worsening from 17.54 to 18.22 after adding SemSeg and Saliency. If expanded task-combination instructions alter the conditioning, this must be stated and separated from parameter forgetting; otherwise the central retention comparison is difficult to interpret.

  2. The counterfactual objective does not by itself establish the claimed correction of harmful experts (Sec. 3.5, Eqs. 13 and 16). With gradients through both losses, the hinge can decrease by increasing the counterfactual loss, rather than improving the original prediction. In particular, the instruction-replacement branch can learn to perform poorly under the substituted instruction. The component ablations show aggregate benefit, but do not distinguish improved factual predictions from degraded counterfactuals, leaving the proposed diagnostic interpretation unsupported.

Questions for authors
  1. Were exactly the same old-task inputs, instructions, evaluation state, and checkpoints used before and after expansion? Please identify what changes the old-task outputs despite the frozen-path claim.
  2. Are the counterfactual prediction losses detached when optimizing Lcf? If not, can the authors report the factual and counterfactual losses separately, or a detached-counterfactual control, to show that reduced regret reflects better factual predictions?
2 comments

We sincerely thank the reviewer for carefully evaluating our work and for recognizing our method's overall performance, parameter efficiency, and ability to adapt to incremental tasks. We are particularly grateful to the reviewer for highlighting the need to further clarify the relationship between the incremental evaluation protocol and the freezing mechanism, as well as potential ambiguities in the counterfactual regret loss. We address the reviewer's two main questions below.

Q1. Were exactly the same old-task inputs, instructions, evaluation state, and checkpoints used before and after expansion? Please identify what changes the old-task outputs despite the frozen-path claim.(Question 1,Weakness 1)

We need to clarify that, for our method, the old-task evaluation protocols before and after expansion are not exactly the same. The two settings use the same evaluation state but different task instructions and different checkpoints, one before expansion and one after expansion; the parameters of the shared modules and existing task modules remain identical. Specifically: Old-task Only stage: We train the model using only old-task data and evaluate its predictive performance using the combined instruction for the old tasks. Incremental Adaptation stage: Starting from the trained old-task model, we freeze HASE, TCM, ICFE, and the routing networks and output heads of all existing tasks, introduce and train only the corresponding task-specific modules for the newly added tasks, and evaluate its predictive performance using the expanded four-task combined instruction. Therefore, the old-task evaluation protocols before and after expansion are not exactly the same.

We emphasize that the purpose of using different task combination instructions before and after expansion is to measure end-to-end performance under actual task expansion conditions. Before expansion, the model only needs to perform the original tasks, so it is evaluated using the combined instruction for the old tasks. After expansion, however, the model should be capable of performing all four tasks simultaneously. We therefore evaluate the expanded model using a joint four-task instruction and examine how well it retains its performance on the old tasks while performing all tasks. Compared with consistently using the old-task instruction, this evaluation better reflects the model's performance under full multitask execution conditions after new tasks are added.

To distinguish prediction preservation under a fixed instruction from performance changes under actual joint four-task execution conditions, we have followed the reviewer's suggestion and added the Old-Instruction Evaluation setting to Table 4 in the revised manuscript. We have reevaluated our model using the following three settings:

Evaluation SettingCheckpointTask Instruction
Old-task OnlyPre-expansion checkpointOld-task instruction
Incremental AdaptationExpanded checkpointFour-task instruction
Old-Instruction EvaluationExpanded checkpointOld-task instruction

The experimental results in Table 4 of the revised manuscript show that, across all ten task expansion settings, the old-task performance is exactly the same under Old-task Only and Old-Instruction Evaluation. Moreover, with the same input images and task instructions and under deterministic inference conditions, the maximum absolute difference in the raw prediction outputs is 0, further verifying the consistency of old-task predictions under fixed evaluation conditions.

In contrast, Incremental Adaptation uses the four-task combined instruction, so old-task performance may change. For example, when SemSeg and Saliency are the old tasks, their mIoU values remain at 70.52 and 63.76, respectively, under the fixed instruction, whereas under the joint four-task instruction, they are 70.11 and 63.43, respectively. This indicates that the changes in old-task performance observed in the original experiments are associated with changes in the task combination evaluation conditions, rather than updates to existing task parameters during incremental training.

We have further clarified the relevant evaluation protocols in Section 4.3 and Appendix A.4 of the revised manuscript and added the fixed-instruction control results to Table 4 to resolve any ambiguity in the original text.

Q2. Are the counterfactual prediction losses detached when optimizing Lcf?L_{cf}? If not, can the authors report the factual and counterfactual losses separately, or a detached-counterfactual control, to show that reduced regret reflects better factual predictions?(Question 2 , Weakness 2)

To first answer the question about gradient settings directly: when optimizing Lcf\mathcal{L}_{cf}, the original method does not detach either the expert removal loss Lt(−j)\mathcal{L}_t^{(-j)} or the task demand replacement loss Ltc′→c\mathcal{L}_t^{c'\rightarrow c}. The Counterfactual Regret Loss remains differentiable, allowing the expert modules to be updated and gradients to propagate to the actor network through the reparameterized expert weights.

We understand the reviewer's concern: the identification and correction mechanism described above does not imply that every decrease in the hinge objective necessarily results from improvements in factual predictions. When gradients are retained for the counterfactual losses, it is indeed necessary to examine whether regret is reduced by increasing the counterfactual losses. We have therefore added the following analyses that directly examine the optimization behavior.

In the Counterfactual Optimization Analysis in Appendix A.5, we use the same batch of 64 validation images to evaluate the checkpoints at epochs 201 and 400, reporting both types of counterfactual regret, the three types of branch losses, and task performance.

The results in Table 10 show that both types of regret decrease for all four tasks, while task performance based on factual predictions improves for all four tasks. In addition, the factual loss and both types of counterfactual losses decrease for all tasks, and within each task, the absolute decrease in the factual loss is greater than the absolute decrease in either type of counterfactual loss. In particular, the loss of the task demand replacement branch, which the reviewer specifically highlighted, also decreases for all four tasks, with no overall increase.

These results show that, for the reported fixed validation subset and the two checkpoints evaluated, reductions in regret are accompanied by lower factual prediction losses and improved task performance, while both types of counterfactual predictions also improve. Thus, the observed results do not support the explanation that "regret is reduced solely by degrading counterfactual predictions overall," and provide direct empirical support for the effectiveness of the expert learning and combination correction mechanism.

Following the reviewer's suggestion, Table 6 in the revised manuscript compares the original full model with the TICR w/ Detached CF Losses variant. In the latter, gradients are stopped for both types of counterfactual prediction losses when constructing Lcf\mathcal{L}_{cf}, while gradients for the factual prediction loss are retained.

The results in Table 6 show that the original model performs better on all four tasks: mIoU increases by 1.82, 1.16, and 1.09 percentage points for SemSeg, Parts, and Saliency, respectively; mErr decreases by 1.18 for Normal; and the aggregate metric Δm\Delta_m increases by 3.26 percentage points. This indicates that, under the experimental settings in this paper, retaining counterfactual gradients helps achieve better final task prediction performance.

We need to clarify that what we refer to as a "harmful contribution" applies only to the current sample, task, and weight configuration: if the loss decreases after an expert is removed and the remaining weights are renormalized, this indicates that the current combination has room for improvement; it does not mean that the expert is harmful to all samples and tasks. During training, the task loss and the differentiable counterfactual loss jointly optimize the experts and the actor, while PPO further optimizes the routing policy. Thus, this mechanism adjusts both expert responses and how they are combined to improve factual predictions, rather than merely reducing the probability of a particular combination, and it does not guarantee that every update will reduce the weight of a particular expert.

Expert removal provides an explicit criterion for identifying harmful contributions within the current combination, while the regret penalty and PPO feedback provide an optimization mechanism for correcting expert combinations. The newly added Tables 10 and 6 provide empirical support through an analysis of optimization behavior and a comparison of gradient settings, respectively. We do not extrapolate these results into a guarantee of monotonic improvement for every update or every sample. However, they support the view that TICR uses counterfactual diagnosis to improve expert combinations and factual predictions, rather than relying on overall degradation of the counterfactual branches.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.