- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of which TTA method to choose when receiving new data. It asks when it even is possible to distinguish the best action to take, under the evidence the selector is given. The paper shows that even a perfect selector cannot always choose the best action, given the evidence.
Critical strengths
-
The paper explores an interesting research direction of when it even is possible to choose the correct adaptation method.
-
There are experiments on a variety of datasets and different selector are used in experiments to compare against.
Critical weaknesses
-
The paper is extremely difficult to follow. It presents a quite simple problem setting and method, in an extremely convoluted way. Extremely many keywords are used and the language is structured very complicated. Formatting is also off in many places. It makes it hard to understand what the main contribution of the paper is.
-
Much of the theory that is presented, is trivially derived from the problem formulation. They also do not seem to provide a foundation for deciding when an adptation method will be helpful and are seemingly detached from the experiments.
Questions for authors
No further questions.
Response to the authors
Thank you for the response and revisions. The new matched-pair regret analysis strengthens the connection between the theoretical formulation and the experiments. In particular, showing that the order-blind selectors perform near the minimum regret achievable under a shared-action constraint helps distinguish insufficient information from limitations of a fitted selector. The matched-regressor comparison and clarification of the RF-versus-Ridge results are also useful.
I appreciate the clearer distinction between the familiar impossibility principle and the claimed TTA-specific contribution. I also recognize that an analysis of information limits need not provide a universal rule for choosing an adaptation method.
My remaining reservation concerns the significance and practical scope of the resulting insights. The strongest evidence comes from controlled reorderings and selected strong pairs, leaving unclear how frequently consequential action ambiguities arise in typical deployments. The natural-data experiments provide a useful boundary case, but SOURCE remains sufficient throughout the tested action menu, so they do not establish a practical need for stream-aware selection in those settings.
Overall, the rebuttal meaningfully addresses my concern that the theory and experiments were disconnected. I am therefore raising my score from 1 (clear rejection) to 2 (weak rejection). My remaining concern is whether the TTA-specific findings provide sufficient insight beyond the general observation that order-blind evidence cannot distinguish streams requiring different actions.