- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduce a novel way of solving the problem of ASK or DEFER which is called the repairability. And a cost-aware router about how to more effectively choose DECIDE/ASK/DEFER.
Critical strengths
- The idea is very novel, the paper does not stop as the three types of decisions, but also move further about the how to repair them.
- The experiments are quite solid.
Critical weaknesses
- I think in general it is a very good paper, but only the worry is that in the evidence repairability is there any theory guarantee in there?
- I feel the idea and way of doing this problem is really nice but only that there is no theory guarantee.
Questions for authors
- In the paper, the author only separate consider the problem of ASK & DEFER, and why the author did not consider to put them together?
- Also could authors provide more guarantee about when the repairability provides improvement.
We appreciate the reviewer's positive assessment of the novelty and experimental design. The reviewer's main concern was the lack of a theoretical guarantee; in the revision, we (i) clarify that ASK and DEFER are jointly selected, and (ii) add a formal analysis characterizing when joint repairability-aware routing improves utility and how estimation error affects a learned value-based router.
Q1. Are ASK and DEFER considered jointly?
Yes. ASK and DEFER are estimated separately because they repair errors through different mechanisms: ASK changes the available evidence, whereas DEFER changes the decision maker. However, they are selected jointly.
In the primary frozen router, ASK and DEFER first satisfy their development-frozen eligibility thresholds; when both are eligible, the router directly compares D^Q and V^H and selects one action (Appendix B.6). In the cost-aware formulation, this joint comparison is written explicitly as
a∗(s)=argmax{0, D^Q(s,M)−cQ, V^H(s,M)−cH}.
This rule applies after the common risk gate and subject to ASK admissibility. Thus, the two repair channels are modeled separately but compete within one DECIDE/ASK/DEFER policy. We revised Section 3.4 to make this explicit.
Q2. Can repairability provide a theoretical guarantee, and when does it improve performance?
Yes, for the value-based formulation, together with a conditional bound for its learned plug-in implementation (Section 3.5; proofs in Appendix C.6). Let
gQ(s)=DQ(s,M)−cQ,gH(s)=VH(s,M)−cH
denote the net values of evidence repair and capability repair, with DECIDE assigned value zero, and let U(π) denote the expected net gain of policy π over DECIDE.
Proposition 1 (Oracle dominance). The oracle joint router weakly dominates DECIDE-only, oracle ASK-only, and oracle DEFER-only because pointwise
max{0,gQ,gH}≥max{0,gQ},max{0,gQ,gH}≥max{0,gH}.
Proposition 2 (When joint repairability helps). The gain over oracle ASK-only is exactly
U(πjoint∗)−U(πASK∗)=E[(gH(s)−max{0,gQ(s)})+].
Hence joint routing strictly improves over ASK-only exactly when capability repair has greater net value than both evidence repair and DECIDE on a set of states with positive probability; the result relative to DEFER-only is symmetric. Joint routing therefore strictly improves over both oracle single-channel policies exactly when each repair channel has the strictly highest net value on some set of states with positive probability.
Proposition 3 (Plug-in estimation bound). For the value-based plug-in router, suppose
maxa∣g^a(s)−ga(s)∣≤ε(s).
Then
U(π^)≥U(πjoint∗)−2E[ε(s)].
Moreover, if the oracle gap between the best and second-best actions at state s exceeds 2ε(s), the plug-in router selects the oracle-optimal action at that state. The proof follows from
gπ∗≤g^π∗+ε≤g^π^+ε≤gπ^+2ε.
These results also hold under a common risk gate: states rejected by the gate take DECIDE under all compared policies, and the analysis applies to admitted states. They characterize the value-based rule rather than the separately thresholded primary router, and Proposition 3 is conditional on the estimation error rather than a finite-sample guarantee (Section 5).
Empirical connection. In hindsight-oracle accuracy, joint routing reaches 75.84% versus 68.20% for oracle ASK-only, illustrating the complementary repair opportunities characterized by Proposition 2. In the primary deployable experiment, joint routing improves over ASK-only from 59.02% to 62.69% (+3.67 pp; 95% CI [1.53, 5.81]; exact McNemar p=0.001831), demonstrating a statistically significant held-out gain for the evaluated Llama model pair.
We appreciate the reviewer's recognition of the paper's strengths and hope the added theoretical analysis addresses the remaining concern about guarantees. We would be grateful if the reviewer would reconsider the score in light of the revision.