- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors appear to study whether Transformers implement gradient descent when performing in-context linear regression. To do so, they compare the loss attained by the Transformer with the loss attained by gradient descent after a variable number of steps. They suggest that for small sample sizes, Transformers can readily exceed the performance of gradient descent trained on a number of steps equal to the model depth, suggesting that the learned algorithm is different. However, their comparison is inconclusive for larger sample sizes when gradient descent attains a lower loss.
Critical strengths
The authors study an interesting topic, and the general direction may hold promise.
Critical weaknesses
I found the manuscript to be excessively difficult to read. The organization of the text is very unusual, and ideas seem to be introduced and referenced out-of-order. The authors also appear to use unusual jargon that isn't defined or motivated. Formal results appear to be injected into the manuscript without much introduction or motivation, further making the manuscript difficult to parse. From the AI use statement, it sounds like the manuscript was written largely or entirely by LLMs, and the haphazard style does seem plausibly consistent with this. The manuscript would benefit substantially from a large editorial pass to ensure the writing and message is coherent.
The central message, as far as I was able to understand, was that Transformers don't seem to be implementing exact gradient descent when trained on in-context linear regression. Their method is sensitive to this conclusion for small in-context sample sizes, but fails to make this distinction for large sample sizes. While an interesting result, it's novelty is unclear, as this conclusion has been exhaustively studied and well established since von Oswald 2023. It remains unclear what additional knowledge was produced.
Questions for authors
None
Thank you for connecting the paper to von Oswald et al. (2023). We agree that departures from ordinary GD, including the GD++ behavior they studied, are not new. Our contribution is a quantitative test of whether any member of a specified family can approximate the transformer’s predictions.
A GD run returns one coefficient vector. Our comparator instead selects the best vector from the entire span allowed by the budget, using the true regression coefficient to make that choice (Eq. 2; Proposition 1). A particular GD run can lose because it was poorly tuned; the transformer’s advantage over this oracle cannot be explained that way. At 22 examples both 12-layer models beat even the budget-16 unpreconditioned oracle (Table 1).
The approximation result goes further. On each model’s reported simultaneous confidence event, every budget-12 unpreconditioned predictor is farther from that model’s predictions than the true target is (Table 11). We agree that the presentation made the central contribution hard to follow. We should have made the distinction between a single GD run and the family-wide comparison clear from the start. The guarantee is about how closely an entire tested family can match the observed predictions; it does not identify the model’s internal optimizer or rule out GD++ in general.
We ask that you reconsider the soundness and overall ratings in light of the family-wide guarantee in Proposition 1 and the approximation-distance bound in Table 11.