- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Proposed: CODA (criticality via Optimal Interventions under decaying agency), a method that permits agentic intervention of the fallback policy, but penalizes interventions by making future interventions less likely to take effect. Ie. ~If I intervene, I better be sure it is worth it, and this underscores the importance of those interventions. Namely,
- This is analyzed/characterized as a sequence of coupled optimal stopping problems
- Then they propose an algorithm corresponding to this perspective
Critical strengths
Strengths:
- This fits nicely into popular frameworks like Lazy-MDP, building off it to penalize overriding but in a clever way of decreasing future agency
- This obfuscates reward-scale tuning for intervention and makes the threshold state-dependent
- The method is principled and budget-aware in comparison with the Q-variance heuristic solutions (Q-value spread, return-variance criticality)
- The analysis and proofs look clear and correct, eg. Prop 5. (On the option-value vs global threshold) Is interesting and appears novel (if I am not mistaken)
- This method can be controlled intuitively it appears from results, by tuning beta (exogenous chain example)
Critical weaknesses
Weaknesses:
- Continuous evaluation is difficult and it seems unsolved totally by the proposed method
- This is as the authors mention because compute scales with K
- This requires random starts, as criticality is trustworthy only with good rho-visitation
- Experimental results show a large overlap with Lazy-MDP
- The experiments are largely confined to small environments where this works well, but neglects more complex tasks where the authors propose this method would be useful
- Even a toy human-in-the-loop study is missing despite being a motivational case
Overall: CODA is a principled and interesting method that seems to work well for simple/small environments. That being said, the method is fundamentally limited in larger/continuous spaces and its evaluation is currently narrow, challenging its practical usage. It seems that if the authors could demonstrate the success in larger environments, they might convince me that additional tricks are needed to make the method work.
Questions for authors
None at this time
Dear Reviewer dWrR,
As the discussion period ends in less than 24 hours, we wanted to check whether our response has addressed your concerns, and whether any questions remain that we can still answer.
In brief: our response addressed the significance issue by explaining the key properties that distinguish CODA from Lazy-MDP and by running a further experiment in a relaxed evaluation setting that showcases CODA's advantage, without requiring any additional training. To address your question about the proposed criticality criterion, we showed how it relates to our framework, especially in the context of Proposition 5, visualized its critical states on CliffMaze, and compared it against the baselines on the budgeted intervention task on UnlockPickup.
Since then, we have also added a comparison with Lazy-MDP on LunarLander, which you can find in our latest reply to Reviewer RzU1.
Thank you again for your review. We would be grateful for any further feedback.