- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This is a well-written empirical paper. it starts with a useful observation that raw hidden-state MST error in OPRD is dominated by late-layer representation scale, while holding a much smaller version of the damage. The paper conducts comprehensive experiments, including rewriting and placebos, and proposes a simple method to truncate the last four layers. OPTD-4 improves pooled accuracy by 3.97 points over the reproduced OPRD baseline.
Critical strengths
- The paper is empirically sound with a clean solution. It motivates the problem well from observations, and the proposed algorithm is straightforward and easy to implement.
- The experimentation is comprehensive. It compares along with several placebos and straightforward weighting.
Critical weaknesses
- The paper is limited to one highly specific model pair, despite covering multiple datasets.
- The splice measurement is not a depth-neutral measure of layer importance. Naturally error will propagate, so under this measurement earlier layers contribute more.
- Based on the previous points, it does not mean that later layers are not important due to the cascading factor. Especially during early learning, the signals from later layers may not be stable or accurate due to compound error. A gradual learning from early to late layers might be more effective.
- In practice, it's hard to determine how many layers to truncate. This seems to be more of an empirical choice.
Questions for authors
See weaknesses. I suggest the author to focus on 1) expanding to another model pair. 2) study learning schedule: fit earlier layers first then late layers 3) how to select N empirically without choosing all.