- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies sequential evidence acquisition from partial information aiming to balance the cost of acquisition and prediction accuracy. They proposed Subset Lattice Distillation (SLD), which build a teacher model using complete information from an offline dataset to learn cost-sensitive exploration and stopping decision. Then distill its knowledge into a student model that operates under partial (observation so far) information online.
Critical strengths
- This paper provided theoretical guarantee for when distilling from teacher model under full information into a student model, which act under partial information so far, preserves good action making. This contribution to the novelty of the proposed algorithm.
Critical weaknesses
- The paper lacks a discussion of the best-arm identification and active hypothesis testing literature. The authors should compare their approach with relevant methods and include appropriate baselines from these areas to better contextualize their contributions.
- The evaluation does not sufficiently compare SLD with other baselines under diverse acquisition-cost profiles. Table 4 reports SLD's performance under cost perturbations that primarily modify one evidence source at a time, which is insufficient to demonstrate its robustness or comparative advantages across different cost structures.
Questions for authors
- Could authors evaluate relevant baselines under a broader range of cost profiles, including scenarios that vary multiple acquisition costs simultaneously and alter their relative rankings.
- MLP (WW+ED) is a highly competitive system, as shown in Table 1. How would the author interpret its result in comparison with SLD? Could WW+ED alone be sufficient to achieve comparable performance?
3 comments
Q3: "MLP (WW+ED) is a highly competitive system, as shown in Table 1. How would the author interpret its result in comparison with SLD? Could WW+ED alone be sufficient to achieve comparable performance?"
Answer: This is a good question. MLP (WW+ED) is a highly competitive low-cost option; SLD improves epidemic-phase prediction and calibration in Table 1. Emergency-department visits are highly informative in this task. SLD improves phase F1 from 0.964 to 0.983 and reduces expected calibration error from 0.073 to 0.014 (Table 1). WW+ED remains cheaper and has stronger binary-alert results (Section 6.2; Appendix C.2, Table 5).
One major contribution is turning complete training records into supervision for acquisition and stopping. SLD achieves higher phase F1 than GDFS at essentially the same acquisition cost and than ACO at lower cost (Section 6.1, Table 1), demonstrating more effective acquisition and stopping.
I appreciate the additional experiment that you have added adapted hypothesis testing and experiments on more diverse cost profile. My concern has been addressed, so I will raise my score.
Thank you for recognizing the theoretical contribution of transferring decisions from a teacher model with complete information to a student model with partial observations. We address the connections to the literature and provide additional comparisons below.
Q1a: "The paper lacks a discussion of the best-arm identification and active hypothesis testing literature."
Answer: Thank you for pointing out these connections. Existing work in best-arm identification [1,2] and active hypothesis testing [3] does not directly handle our problem setting. Specifically, best-arm identification repeatedly samples arms to identify the highest mean rewards; our task predicts a case's label and reveals each evidence group only once (Section 3). Classical active sequential hypothesis testing requires known observation distributions, which our setting lacks. SLD instead scores evidence subsets in complete records, uses backward planning to construct multi-step queries and stop targets, and distills them into a policy that sees only acquired evidence (Sections 4.2-4.3, Equations (5)-(7)). We will cite these papers and add these connections and distinctions to Section 2.
References
[1] Garivier, A., and Kaufmann, E. Optimal Best Arm Identification with Fixed Confidence. COLT, PMLR 49:998-1027, 2016.
[2] Kaufmann, E., Cappé, O., and Garivier, A. On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models. Journal of Machine Learning Research, 17(1):1-42, 2016.
[3] Naghshvar, M., and Javidi, T. Active sequential hypothesis testing. The Annals of Statistics, 41(6):2703-2738, 2013. DOI: 10.1214/13-AOS1144.
Q1b: "The authors should compare their approach with relevant methods and include appropriate baselines from these areas to better contextualize their contributions."
Answer: These are good suggstions. Following your suggestions, we evaluate the adapted active-testing baseline on a synthetic benchmark (also following Reviewer ZYko's suggestion) because it allows us to compute the optimal acquisition and stopping decisions exactly. Against this reference, SLD achieves lower mean objective regret and higher agreement with optimal actions, as shown in Table R1 below. The methods share the predictor and acquisition costs, and their online decisions rely solely on acquired evidence. We will include these results and the adaptation's query and stopping rules in the revision.
Table R1. Comparison with adapted active hypothesis testing
Results averaged over three runs. Regret is excess expected prediction loss plus weighted acquisition cost relative to the exact online optimum, which uses only acquired evidence.
Q2: "The evaluation does not sufficiently compare SLD with other baselines under diverse acquisition-cost profiles. Table 4 reports SLD's performance under cost perturbations that primarily modify one evidence source at a time, which is insufficient to demonstrate its robustness or comparative advantages across different cost structures." "Could authors evaluate relevant baselines under a broader range of cost profiles, including scenarios that vary multiple acquisition costs simultaneously and alter their relative rankings."
Answer: Good suggestion. Following this suggestion, we ran additional experiments and showed that when multiple acquisition costs and their relative rankings change together, SLD achieves higher mean phase F1 at lower mean cost than both ACO and GDFS in each new setting (Table R2).
Table R2. Adaptive acquisition under joint changes in costs and their rankings
Results averaged over three runs with a shared predictor and prices within each profile. SLD replans and retrains its policy under each profile (Section 6.4). Cost vectors (emergency-department visits, hospital admissions, laboratory positivity, source context): default (1.0, 1.5, 1.2, 0.4); Joint A (0.4, 1.2, 1.5, 1.0); Joint B (1.5, 0.4, 1.0, 1.2). The full acquisition cost is 4.1 across all profiles.