- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper hypothesizes that neural networks trained on dimensional physical data may learn hidden representations that capture dimensionless relationships, and empirically investigates this hypothesis.
Using the Buckingham-Pi theorem, the authors formulate a linear map from dimensional inputs () to dimensionless variables () in logarithmic coordinates. They analyze the learned neural representation by decomposing its Jacobian using the orthogonal projector onto . Specifically, the authors measure the energies of the projected components as and , and then compute the physical share score as a fraction of the physical component, . This decomposition has a natural interpretation from the quotient perspective of the Buckingham-Pi theorem: represents directions in that change , capturing nondimensional physical changes; on the other hand, its orthogonal complement, , represents directions that preserve , corresponding to mere dimensional scale variations of the same physical law. Accordingly, and quantify the representation's sensitivity to physical and scale variations, respectively.
Importantly, if the neural representation depends only on the scale-invariant dimensionless coordinates, i.e., , the chain rule gives . Consequently, in this case, $ E_\mathrm{scale}$ vanishes, leaving only and accordingly . The authors train FNOs, CNNs, MLPs, and Transformers on raw dimensional data from seven different physical systems and measure changes in physical share before and after training. They find that physical share increases across all seven systems, reaching 0.9-1.0 in many cases. They further validate this finding through various controlled experiments, and also show that this alignment extends beyond the infinitesimal Jacobian level to finite input transformations.
In the subsequent experiments, the authors investigate whether this phenomenon simply reflects different architectures learning similar latent coordinates. They find that linear mappings between hidden representations do not consistently enable successful model transfer. Based on these results, the authors argue that the observed convergence toward dimensionless, scale-invariant representations reflects the underlying physical relationship rather than bias toward a common latent coordinate system.
Critical strengths
This paper proposes an interesting hypothesis in scientific machine learning. Inspired by the Platonic representation hypothesis and the Buckingham-Pi theorem, the authors hypothesize that neural models trained on raw dimensional data may learn internal representations organized around dimensionless coordinates. Through carefully designed experiments based on a quotient-space perspective, the authors provide empirical evidence supporting this hypothesis.
Critical weaknesses
W1. Overall, I find the hypothesis and supporting evidence presented in this paper interesting. However, I believe the paper would benefit from improved readability, particularly in how the experiments are presented.
While the authors relatively clearly explain their motivation, hypothesis, and the mathematical tools developed to test it, the presentation of the experimental designs and results is somewhat vague. For example, Fig. 1 occupies more than half of page 3 but is never discussed in the main text. Moreover, its relevance to the central hypothesis is not immediately apparent.
Fig. 2 presents the main experimental results and receives some explanation in the text, but the purpose of each panel is not easy to follow. For instance, Fig. 2 (c) shows that physical share remains elevated even after removing the clock direction, but the motivation of this control experiment is not clearly explained. I find the control experiments in Fig. 2 (b-d) well designed, which makes the lack of clear explanations unfortunate. Similarly, Fig. 3 (a-c) receives very little explanation, making it difficult to understand exactly how the experiments were conducted without considerable inference.
Given that there is still substantial unused space within the page limit, I suggest that the authors more clearly explain the purpose of each experiment, the control variables, and how the results support the central hypothesis.
W2. As the authors acknowledge in Section 6, this paper presents empirical evidence supporting the proposed hypothesis but does not explain the underlying mechanism. Moreover, the current version does not demonstrate how the observed phenomenon could lead to practical benefits. I do not consider these as particularly serious weaknesses. Nevertheless, while the paper deserves recognition for its novelty in reporting an interesting phenomenon, its immediate practical significance and contribution to a deeper theoretical understanding remain limited.
Questions for authors
Q1. I understand that the purpose of Section 5 is to distinguish physical organization from prediction accuracy. However, I am curious about the (empirical) relationship between these two properties. For example, what would we observe if one plotted against prediction error or across different models and tasks? Would there be no clear correlation, or might some systematic relationship emerge? Since many models achieve physical shares close to 1, such a relationship might be difficult to observe. However, could a more detailed analysis of individual runs, different initializations, and training checkpoints reveal meaningful trends?
Q2. It would also be interesting to compare models trained on raw dimensional inputs with those trained on explicitly dimensionless coordinates. The latter naturally enforces dimensional scale invariance. Do the two approaches yield learned representations with similar geometric structures?
Q3. Buckingham-Pi coordinates can be interpreted as quotient coordinates under dimensional scaling symmetry, which is a particular class of Lie group symmetries. If the governing physical laws possess a larger Lie group symmetry beyond dimensional scaling, what properties would the learned representations exhibit? Would they remain invariant only under dimensional scaling, or would they also capture the broader Lie symmetries?
Q4. This may already be explained in the Appendix, but I would appreciate more details on the experimental setup. Specifically, if the training data consists of raw dimensional quantities, the target outputs may also depend on dimensional scales. In that case, a scale-invariant hidden representation would not be sufficient to produce the correct outputs (unless the model has an explicit scale-dependent output head). Could the authors clarify whether the learning objectives and target outputs in each task are scale-invariant or scale-equivariant, and how dimensional scaling is handled in the model?
Q5. Both the orthogonal projection and the energy norm depend on the choice of metric. How robust are the findings to this choice? Have the authors examined whether the results hold under metrics other than the Euclidean metric?
Minor comment: Although Section 6 is titled "WHAT EMERGES ACROSS PHYSICAL SYSTEMS?", the authors acknowledge that "Why this organization emerges remains unresolved." Therefore, I suggest renaming this section to something more appropriate, such as "Limitations."
Thank you so much for your positive (and also critical) feedbacks!
Firstly, you are right about the readability of the experiments. Section 2.1 now discusses Figure 1 as motivation and distinguishes its illustrative networks from the new 72-predictor study. Section 2.2 explains the Burgers example before the measurements: joint rescaling preserves the normalized solution, while changing viscosity alters it. Section 2.3 explains why we compare with initialization, match input distance and hold the clock fixed. Holding the clock fixed asks whether selectivity extends beyond elapsed progress. Appendix C.1 specifies the cases, controls and uncertainty. The figure numbers have changed with the new study.
Furthermore, we agree that the submission left its consequences unclear. Section 3 now checks recovered relations against independent physical calculations, using finite Burgers paths and an anomalous exponent within a prescribed family. Section 4.3 and Appendix F.3 connect scale sensitivity to finite generalization. Output sensitivity ranks generalization as closely, so the hidden measure has no demonstrated predictive advantage here. A unique learning mechanism remains unresolved.
A1: We find a strong within-run association and a weaker one across settings. Following your suggestion, Appendix F.4 compares 48 Burgers predictors. Median within-run rank correlations between error and contrast exceed 0.98 at every measured layer. Here a more negative contrast means stronger selectivity: error and contrast decrease together. Across selected checkpoints, the correlations are 0.63 for FNOs and 0.54 for Transformers. In Table 22, width-256 Transformers at learning rates 10⁻³ and 3×10⁻⁴ have median errors of 2.05% and 2.23%, with last-hidden contrasts of −1.13 and −0.76. Thus similar accuracy can accompany different selectivity. These validation-selected checkpoints establish a Burgers comparison, not a cross-task law.
A2: They show related scale selectivity expressed at different depths; we have not established identical complete hidden geometries. We added matched Burgers predictors with raw parameters, logarithms, dimensionless groups plus scale, and groups alone (Section 4.3; Appendix F.2). With logarithms or groups plus scale, dependence on scale is largely removed by the first affine stage. With raw inputs it decreases through nonlinear layers. Groups alone enforce invariance by construction. The encoding also changes the coordinate channel and training normalizers.
A3: Some are, to different degrees. Sections 4.1 and 4.2 (Appendix E) add time re-anchoring, translation and Galilean boosts of periodic Burgers dynamics. In the same twelve evolved-profile predictors, hidden layers attenuate scaling more strongly than time re-anchoring, although both preserve the target. Accurate outputs approximately respect both. Translation and boost responses depend on architecture and layer. The finite tests use a specified feature-action class; they do not establish a full Lie-group representation or composition laws. We exclude the projective transformation because it does not preserve the periodic problem.
A4: Your concern is correct: a predictor must retain any scale needed for its output. Section 2.3 and Appendix A clarify that the main study's selected similarity moves preserve its prediction targets. Appendix F.5 adds a separate amplitude test: the network either predicts the dimensional Burgers field or predicts its normalized field with amplitude restored outside the network. Both use the same physical loss. FNO last-hidden amplitude-response shares are 0.16 to 0.17 and 0.005, respectively. Invariance must therefore be assessed relative to the actual target.
A5: The original feature-rescaling checks did not establish metric independence. Appendix B.4 gives the revised-cohort tests: on Burgers and the pendulum tasks, sampled channel transformations with condition numbers up to 10 retain more than 92% of the training-induced changes in similarity share and weak-direction angle; standardization retains more than 90% (system means). General invertible transformations can change these measurements. But, the claim is limited to these transformations and populations.
Thank you again for recognizing the potential in our research. We are grateful for your review and it has allowed us improve the manuscript. We are happy to answer more questions!