- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
StatOrch trains selectors on offline tasks to choose statistical methods from summaries of unlabeled target data. The executed pipeline adapts a predictor, estimates class proportions, and selects a classification rule, with a fixed fallback when expected gains are small. Other experiments separately evaluate measurement choices and a broader method pool.
Critical strengths
- The problem is stated clearly. Inference on an unlabeled target becomes a choice among prior correction, adaptation and acquisition, each tied to the evidence that supports it, with the current procedure kept as a fallback.
- The order of execution handles the dependency between stages well. Adaptation replaces the predictor and the reference a quantifier needs changes with it, so the proportions are recomputed afterwards and the joint objective is reported for the whole chain.
Critical weaknesses
- The added value of refreshing the evidence between stages remains unestablished. Tables 3 and 34 do not separate the two: the seen joint loss is 0.3309 for stagewise and 0.3300 for one-shot, the paired interval spans zero, and on unseen mechanisms stagewise shows no advantage. The reported comparisons therefore do not demonstrate a gain over one-shot selection that would justify the additional stages and evidence refreshes.
- The adaptation method is chosen by a criterion that the paper does not show to be aligned with the final objective. Equation (6) ranks candidates by a stage-local proxy, EMQ proportions plus known-class Macro-F1, which the paper states is not the terminal objective, while the later stages choose their own proportion estimator and classification rule. Whether the proxy's ranking agrees with the ranking under the final joint loss is not reported.
Questions for authors
- In Tables 3 and 34, is there a split or task group where stagewise selection gains enough over one-shot to justify refreshing between stages? The seen joint losses are 0.3309 and 0.3300.
- Do the recorded pipelines rank the adaptation candidates the same way under Equation (6) and under the final joint loss? Where the two differ, how large is the final gap?
- In Table 3 the greedy variants give 0.3234 for stagewise and 0.3237 for one-shot. Does that difference fall inside the paired interval, and does the ordering hold across the four fitting seeds?