- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study focuses on the question of whether the models learn the rule behind a task or just the answers. The task they use is composing 3D rotations and the rule which is the Lie bracket captures how much the order of rotations matters. They tested 8 models and they found that the untrained (not fine-tuned) models do not show signs of learning the rule while LoRA fine tuning installs partial traces in all but one of the models. They use a baseline that ignores the rule (order blind predictor) and they argue that if a model is better than the baseline then they may be using the rule but even the fine tuned model does not beat the (strong) baseline beyond noise. Authors claim that 3 interpretability checks come out (by mathematical necessity) positive.
Critical strengths
The authors make sure that they compare the results to cases where the right answer is known and make sure to take back any claim that does not hold. This makes their results to be trusted more. In a steering style test the correlation was 0.96 even when there was no model which is a useful warning for future studies.
Critical weaknesses
I did not see a control where the rule is not correct, but the order still plays an important role. The claim they make about llama-3-8B is only based on one model and a single run. The part about learned, but not used is not sufficiently supported, considering that defined tune model learns only a little of the order dependant part so I am not surprised that it is not using the rule. I would want to see how the answers would change if they remove the rule from the model.
Questions for authors
- Have you tried measuring the rule score when the rule is scrambled (or replaced with other algebra) and in the case that order does not matter.
- I want to know if you’re results would still hold if the move names do not share word pieces?
- Have you tried removing the rule from the model? How does the results change?