- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors study a setting where an agent failed to complete some task, and an auditor (called ``allocator") tries to reveal which decisions led to the failure under a limited budget. In particular, they define repair value : the improvement that the best valid alternative gives over the executed action, divided by the effort of evaluating the candidates and Capture which is the ratio of repair value over an oracle measured improvement which has upfront access to all outcomes and operates on the same budget. They propose CIA, an algorithm to estimate repair value and do a series of experiments to evaluate its efficiency under different scoring functions (functions that represent the value of an action).
Critical strengths
- The overall motivation of the problem is strong: failure of an agent that tries to repair it (itself or with external help) by examining its decisions and alternatives is a reasonable process.
- There is a variety of factors considered in the designed experiments.
Critical weaknesses
-
The exposition is weak. The paper does not have a coherent order of presentation of definitions, model assumptions as well as explanatory examples. The effort spent on formalizing the setup is minimal. There is presentation of the units and domains of the quantities and functions introduced in Section 2.
another example:
In page 2, when basic setup is formalized there is this sentence the simulator sets gi when it constructs the alternatives, so positive score differences without a designated valid repair receive zero value. The term ``simulator" is not defined before and it is unclear what is does.
Section 3 also comes absurdly; how does information about prediction of repair opportunities relate to the set up presented on section 2? What do we need to predict? -
The proposed methods are rather simple ; a combination of nearest neighbors search with standard ML techniques as ridge regression. The gains on Capture as presented on Table 1 over nearest neighbors are also not substantial.
Questions for authors
-
In the setup of the problem after taking action a the allocator is revealed the scores for all actions in . Why is that a reasonable assumption in practice? More often than not taking an action has binary feedback: we either gained the outcome we expected or not. In addition, generating the scores of an action is more often than not, a costly procedure itself. So I don't see how or why an allocator would receive such strong information. Please provide relevant examples or justify the assumption.
-
On page 2, on the denominator of capture there is a term , what is this term exactly?
-
What is the main benefit of your method vs NN?
-
Since this is about agent failure, is there an experiment you can do using an actual agent executing alternative actions, rather than relying primarily on simulator-defined outcomes? If not why?