- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether mechanistic knowledge of a model’s internal representations can be used to improve task performance through direct geometric supervision. Using Llama-3.1-8B, the authors study two tasks: month arithmetic, where relevant variables are represented by Fourier features, and linebreaking, where character count follows a helical structure. In both cases, a LoRA trained to align the internal representations with an idealized geometry improves performance without any output-level task loss.
Critical strengths
The paper presents an interesting idea that uses mechanistic knowledge of internal representations to construct a training objective rather than relying exclusively on output-level supervision. The results on month arithmetic show that this can work well.
Critical weaknesses
My biggest concern about the paper is that the evidence provided supporting the approach is preliminary. In fact, the evaluation is limited to two highly structured tasks on a single model. It is therefore unclear how broadly the approach applies, especially when the relevant variables or their geometric structure are not known in advance. The authors really need to expand their evaluations to include more tasks and especially a wider range of models to fully showcase the potential of this approach and how it stands compared to fthe standard ine-tuning.
Questions for authors
- Have the authors explored this approach beyond the one Llama model considered?
- What is the justification for these particular tasks and were there any exploration beyond them?
2 comments
Response to Rebuttal
Thank you for the rebuttal. I understand the premise of the authors but it is hard to judge the usefulness of such understanding given that the proof-of-concept is limited. In my opinion, the paper would really benefit from increasing the breadth of the tasks investigated and the family of models involved beyond the preliminary results, which will make it stronger. With that in mind, i retain my current recommendation.
Response to Reviewer tDAY
Thank you for your review, we’re glad you thought our work was interesting. Our argument is not that this is a robustly supported method for training models that should be used in place of standard fine-tuning. Instead, we wanted to provide a proof-of-concept for how understanding models could possibly lead to us improving them in future research. We focus on two tasks that were deeply investigated by prior research, and leveraging that prior work to test out our new idea about using neural geometry as a learning signal.
We’ve taken this opportunity to sharpen our title, abstract, and introduction to make it clear what our contributions are and the scope of our experimental work! We have posted an updated version of the paper in response to reviewer feedback, and discuss your specific questions here:
Thank you for raising this point. The reason we chose these two tasks (months arithmetic, and in-context linebreaking) is because they have been deeply investigated in prior research, and are tasks where prior work provides a detailed account of relevant variables, their geometric structure, and that geometry’s algorithmic role. Scaling our approach to more tasks is very time-intensive, as it relies on detailed mechanistic accounts of causal variables on new tasks—the scope of an entire other research paper. We have updated the introduction and discussion to make it more clear that our method depends on descriptive accounts of neural geometry, and to clarify the focus of our paper and contributions.
We focus on a single model (Llama-3.1-8B), as this is where prior interpretability research has provided a detailed mechanistic account of the month arithmetic task, and show results for the linebreaking task in the same model. Because our method relies on existing, detailed mechanistic descriptions of the geometry of task variables, extending it to other models that may or may not implement the same mechanism is not always straightforward. However, we have run preliminary analyses on two additional models (OLMo-3-7B and Gemma-2-9B) for the linebreaking task, where we find similar helix-like representations of the \texttt{character-count} variable. Applying the same kind of geometric fine-tuning to these models improves over the base model, but does not achieve as low a linebreaking error as standard LoRA fine-tuning.
As with Llama-3.1-8B, most of the gap is in the low in-context-example regime: in Gemma-2-9B, the geometry LoRA matches the task LoRA once the model has seen about 8 in-context examples (4.5 vs. 4.6 characters), while in OLMo-3-7B a gap remains (10.6 vs. 6.7). These preliminary results report the best configuration from a small hyperparameter sweep for each model.