- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines how the choice of downstream executor affects the evaluation of an intervention. The authors distinguish three levels of impact: changes in the measured effect size, changes in the ranking of intervention settings, and changes in the setting ultimately selected. Experiments in sound event detection (SED), language modeling, image segmentation, and speech recognition demonstrate that these phenomena can occur.
Critical strengths
- The three-level framework provides a useful way to characterize how downstream executors affect the evaluation of interventions.
- The paper examines executor-dependent effects across a diverse set of tasks and output types.
Critical weaknesses
- Although the experiments cover multiple tasks, they offer limited depth within each one. The tasks use different models, intervention families, and executors. Together, the results show that executor-dependent effects can occur, but they do not establish how often they arise within a given domain or how consistently they appear across intervention methods. The current evidence does not rule out the possibility that the observed effects are specific to the chosen models, interventions, or executor combinations.
- The strongest evidence for changes in the selected intervention setting comes from SED. Object detection shows a small local reversal in the ranking of settings, whereas the Qwen3-8B and CTC experiments change the measured intervention response without changing the preferred intervention coefficient. Thus, the experiments support the existence of selection changes but provide limited evidence about their prevalence or practical significance across models and tasks. This limits the support for the paper’s broader conclusions about intervention selection.
Questions for authors
In the Qwen3-8B experiments, did varying the executor produce any local reversals in the ranking of intervention settings or changes in the setting ultimately selected? Can these phenomena be observed across different pretrained models?