- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper develops behavioral models for agents performing tasks in terms of a skill basis. In this way, the approach can produce a set of coordinates the collectively represent the performance of an agent on a task, in such a way that these coordinates correspond to a notion of skill over the domain. In this way, the paper is able to make more fine-grained distinctions among agents, rather than relying solely on aggregate performance measurements. The paper shows that even with this decomposition into an underlying basis of skills, its predictions in terms of these skills are almost as effective as a pure prediction strategy that does not have a skill-coordinate structure.
Critical strengths
The use of a skill basis represents an appealing alternative relative both to pure performance measures and to embeddings whose coordinates don't have a natural interpretation in terms of skills. In contrast, a representation via skills make it possible to better interpret an agent's performance to assess which aspects will carry over to new tasks.
Critical weaknesses
As the paper describes in its section on related work, many past approaches have also sought to decompose an agent's performance into separate components, which can be thought of (explicitly in approaches, implicitly in others) as a set of skills. The paper does not give evidence that its approach provides raw predictive improvement (nor was this its goal), but it also does not really propose a set of criteria under which its particular way of inferring skills represents an improvement over other existing ways of inferring skills. Given this, the paper needs to do a better job providing some conceptual dimension along which its approach would be a desirable thing to try relative to the other approaches that already exist.
Questions for authors
It is entirely reasonable that your method does not improve over others in raw prdiction performance, but there should be some dimension (even if not this one) along which your method has advantages relative to the large body of previous work on representing distinct skills. Can you make clear on what dimensions we should view your approach as conferring benefits relative to this previous work?