- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies why an accurate diagnosis of a system mismatch may still be insufficient for choosing an effective intervention. It treats diagnosis as a compression of information available before the decision and characterizes when that compression preserves attainable intervention value. The analysis decomposes improvement over the best fixed action into representation loss, decision-rule deficiency, and delivered value. For binary actions, it identifies cancellation between opposing action advantages within diagnostic categories as the mechanism of compression loss. The proposed Counterfactual Adaptation Audit links empirical claims to their information requirements, identifying contrasts, and evidential status. A controlled adaptation study demonstrates accurate diagnostic recovery and source-dependent intervention efficacy, with a source-matched rule improving on the best fixed intervention. However, the measured diagnostic becomes available only after a candidate action begins, leaving its pre-decision value unidentified. The study therefore distinguishes recoverable information, decision-relevant information, and information available in time to guide action.
Critical strengths
-
The finding of the paper is overall interesting, which addresses decomposition that separates representation failure from rule failure. Sections 2.2 to 2.3 distinguish value lost through compression from value still available but unused by a particular rule. The binary formula explains how a diagnosis can merge states that favor different interventions. This distinction is useful because poor decisions alone do not show that the diagnostic representation lacks useful information.
-
The paper is careful about what its evidence identifies. CAA separates privileged source information, post-action measurements, and admissible pre-decision information. Appendix E gives two models with identical observed logs but different required repairs. This is a persuasive illustration of why recovery accuracy and intervention outcomes cannot identify which part of a system needs to change by themselves.
Critical weaknesses
-
The experiment does not establish the compression loss that motivates the paper. No legal pre-decision pair and is evaluated through . The residual-arm scalar overlap in Section 5.4 establishes only that a single threshold cannot separate the recorded sources, as the authors acknowledge. The main positive result instead concerns privileged source information, while the recovered diagnostic is acquired after action selection. Thus the empirical study does not demonstrate a case in which diagnostic compression destroys otherwise available decision value, leaving the central theory-to-experiment connection incomplete.
-
The technical advance beyond existing decision-value results is modest. Section 6 and Appendix E explicitly connect the results to feature-gap regret, common optimal actions, and conditional sign-change criteria. Equation (6) is an additive decomposition rather than a new identification result. The main contribution is consequently the CAA synthesis and its reporting discipline. However, CAA does not supply an estimation procedure that identifies its loss terms or a repair from the partially observed record. Further clarification is recommended on the methodological advance beyond organizing established distinctions.
Questions for authors
-
In Section 4, what observations constitute , and at what exact point is the initial value chosen? Is any concrete pre-decision compression evaluated?
-
What update equations, parameter bounds, and fitting budgets define and in the primary decision record? Are their adaptation and evaluation episodes separate?
-
What substantive conclusion does Proposition 3 add to the prior conditional sign-change criterion discussed in Appendix E, beyond accommodating ties and almost-sure statements?
-
For Equation (17), what is the interval under resampling of the five recurring seed blocks with nominal-reference uncertainty propagated? Does the corresponding raw-utility interaction support the same conclusion?
2 comments
Thank you for the detailed response and additional experiment. My main reservation is that the diagnosis is imperfect, with source-label-disagreeing cases supplying the net gain. This leaves the motivating claim that a correct diagnosis can still discard essential decision information empirically unestablished. The concentration of the gain at the −3% calibration level also limits evidence of robustness beyond the specified mixture.
I also retain concerns about the methodological contribution. As acknowledged in the response, the decomposition and sign-change criterion largely follow established results, while CAA organizes evidential requirements without identifying missing counterfactuals or providing a repair procedure. The revisions strengthen the empirical example, but the broader technical advance remains modest.
Response to Reviewer yLLi
Thank you for the detailed questions, especially about when the measurements become available and how the interventions are updated. Those details are central to making the theory–experiment comparison legitimate. We have made them explicit and added F3 as a separate post-review experiment; the original K0 results remain outside its confirmation.
Q1 — Information available at the decision. F3 pays for one common A0 transition and selects a complete program at t=0.05 s. The legal packet Z contains measured gyro and encoder yaw rates, left and right wheel commands, estimated heading, measured wheel-distance difference, interaction index and time. Geometry, controller constants and remaining budget are shared public configuration. The selector does not receive injected source or severity, encoder calibration, seed, true pose, reward or future outcomes.
The fixed diagnosis M is observation-side when g−e>0.025 rad/s and dynamics-side otherwise. The frozen witness additionally uses re=e−(R/L)(wRcmd−wLcmd), with R/L=0.25 and cutpoints −0.025, 0 and +0.025 rad/s. In ascending residual bins, its choices are (A0, replay-A1, encoder-only, encoder-only) for observation-side and (A2, A2, A2, A0) for dynamics-side (§3; Figure 1, p. 4; Appendix G).
Q2 — Updates, bounds and scoring. A1 updates its bias estimate by
bt+1=bt+0.08(gt−bt−et),b0=0.
Heading integrates 0.85(gt−bt)+0.15et, with the new bias estimate affecting later steps. There is no explicit hard bound on b or retrospective heading correction. Replay-A1 applies this ordinary update to the acquired packet once, without another sensor step.
For A2, ht=clip(ψt,deg/45,−1,1), ϕt=(1,ht) and at=tanh(θt⊤ϕt). Its update is
θt+1=0.999θt+0.05×0.3ht(1−at2)ϕt.
The two parameters start with normal SD 0.01 and have no hard box or norm bound; the executed noisy action is clipped to [−1,1] and total commanded yaw to ±1.5 rad/s. This is a heuristic update, not a reward-gradient learner with a convergence guarantee. F3 scores all 2,000 transitions, including the paid prefix, subsequent adaptation, resets and censored final episode. The original endpoint study instead used 2,000 adaptation steps followed by separate locked-parameter evaluation episodes (Appendix E, p. 16; Appendix C, pp. 13–14).
Q3 — Novelty and the missing empirical link. We agree that the additive decomposition and binary sign-change condition draw on established information-value and conditional-effect results; the treatment of ties is a clarification, not a new theorem. CAA organizes which comparisons support which claims, but does not estimate missing counterfactuals (Appendix F, pp. 17–18). The new evidence is an explicit lawful witness compared with all 25 diagnosis-only initial mappings over the same five programs. Its empirical class gap is +0.568464474 and the frozen one-sided lower bound is +0.319293 on the population contrast, under the specified computational protocol. As VZ≥V(πZ), this lower-bounds loss from the fixed diagnosis without estimating the optimal full-packet policy (Figure 3, p. 6; Table 1, p. 7; Appendix G).
Q4 — Original five-seed uncertainty. The normalized interaction is 2.010958, with descriptive joint-bootstrap interval [1.907529, 2.105970]. The raw-utility interaction is 18.237706, with interval [18.032027, 18.429594], and has the same sign. We enumerate all 55=3,125 ordered resamples of the five recurring seed blocks, deduplicate source-labelled nominal records once, and recompute the nominal reference within each draw while keeping seed multiplicities. The 90 source/severity records are not independent units. These historical descriptive intervals are distinct from the F3 confirmation (Appendix C, pp. 13–14).
The F3 finding remains limited to this imperfect diagnosis and the declared simulator mixture. It does not establish insufficiency of a perfectly accurate diagnosis, broad mixture robustness, or hardware safety (§5; Appendix H).