- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes representation dependence (RD), which is Pearson correlation or mutual information computed from fixed probes, in order to monitor the neural network training. Experimental results demonsrate the effectivenss for forecasting overfitting and accuracy.
Critical strengths
The concept of representation dependence such as Pearson correlation or mutual informatoin is interesting. Using the representation dependence between fixed probes for monitoring the neural network training is the novelty of this paper. Experimental results demonsrate the effectivenss for forecasting overfitting and accuracy.
Critical weaknesses
Discussion and analysis regarding experimental results are insufficient.
- There is no result of MI without reason in Figure 4. It would be better to include the result of MI. The effective rank is better than RD_Pearson but there is no reason or description of it.
2.In Table 2, Activation SD (L16) is better than RD-Pearson or RD-MI in Template mixing. The authors should discuss and analyze the result.
-
In Figure 5, there is no result of gradient norm. It would be better to include the result.
-
In Figure 5, Mean |r| is better than Mean r, but the reason remains unclear.
-
Although effective rank worked well in Figure 4, it was not used in other experiments. It remains unclear how the authors select the method in each experiment.
Questions for authors
Please respond to the Weakness I pointed out.