- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper shows that by comparing the performance with and without certain text-based skills, one can select the set of skills that are most helpful to get higher performance on simple benchmarks, and exclude skills that hurt performance on these benchmarks.
Critical strengths
The high-level idea makes sense, is clearly explained, and looks like the sort of approach that is not very common while looking like it ought to work (given the right context, which I am not convinced the paper is studying).
Critical weaknesses
The evaluations are very toy and simple. These tasks do not capture the sort of relatively big and open ended tasks that skills are mostly meant to help with. These are likely tasks that are already saturated by frontier models, and I find it suspicious that some skills are measured to be net-helpful. The paper's Figures do not clearly show the magnitude of the uplift nor what kind of skills result in any uplift on these likely saturated tasks.
The paper has relatively quantitative and concrete qualitative results showing what the approach would look like in practice. It also does not present comparisons with simple baselines (e.g. like asking an LLM if a skill looks helpful or not).
Questions for authors
- Do the helpful skills actually provide an uplift? What do they look like? Why are these strong LLMs failing simple tasks?
- What do the harmful skills look like? Is it trivial to notice these are harmful?
- How would the method perform on more realistic evaluations with either harder tasks or simply more realistic and messy real world settings? I am especially interested in settings where harmful skills are likely to emerge naturally.
Minor
- the lower threshold should likely be <0, since many skills likely reduce performance a tiny amount on most tasks but are extremely useful on some (potentially in ways which outcome based measurements don't capture). Do you agree?
- Could you avoid calling models "Opus"? It would be better for the reader to be able to parse the Figure and get the actual version of the model without having to read the section detailing what version of the model is studied.
- I suggest you rename "harmful" to "performance-reducing", or a similar expression that conveys there is nothing malign about those skills.