- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a synthetic benchmark, SparseWake, for studying whether an underwater robot can estimate the state of nearby fish using sparse local flow measurements. The main challenge is the multi-source case, where several fish contribute simultaneously to the measured flow. The benchmark is carefully constructed and provides useful source-level information that enables controlled analysis of interference and localization failures. The authors evaluate the effects of temporal history, sensor configuration and measurement noise using several baseline architectures. The results show that most errors arise from far-away sources rather than localizing to the wrong source.
Critical strengths
The common prior task is very well motivated and carefully designed. Instead of sampling a privileged target, all sources come from the same distribution, and the selection happens afterwards. The source decomposition is also useful for diagnosis and is able to generate experiments that are difficult to perform with real data.
Critical weaknesses
One major weakness is that the authors do not attempt to transfer to real data. All train/test observations are produced by the same physics-informed wake generator. Evaluation of an out-of-model flow would substantially strengthen the contribution of the paper, but the authors explicitly state that such experiments are not available.
The main empirical result that most of the localization errors come from predictions far from every source, rather than the model selecting the wrong fish, is acknowledged but left unresolved. Utilizing a multi-hypothesis baseline will allow to draw conclusions about whether a model failure is due to intrinsic ambiguity or the claimed conditional averaging.
Questions for authors
Look above
6 comments
Response to Review Comments 2/2
2. Can multi-hypothesis prediction explain the far-from-all failures?
Following your suggestion, we added a multi-hypothesis comparison that allows several candidate locations for the nearest target. If we understood your suggestion correctly, is testing whether forcing an estimator to return one position contributes to the far-from-all errors. Besides, we also expanded the spatial diagnostics and evaluated joint source reconstruction. The three analyses address different parts of this question.
(a) We tested multiple candidate target locations.
We implemented the multi-hypothesis baseline with a five-component Gaussian-mixture head. Each component provides a candidate location with a predicted weight and spread. We compared it with matched point-regression and single-Gaussian heads on each of the same three frozen MLP encoders, giving nine newly trained heads. All heads use the same learned features and nearest-target position supervision, and we selected them using validation data and evaluated them on both common-prior test sets. This comparison helps isolate the effect of the output head.
The table reports the additional test set. Fish counts include the observer, and where L is the reference source-body length. Position metrics compare the point head with the mixture mean, and negative log-likelihood (NLL) compares the Gaussian with the mixture density. Lower RMSE and NLL, and higher hit rates, are better.
| Fish | RMSE / L: point → mixture mean | Hit within 0.25 L: point → mixture mean | NLL: Gaussian → mixture |
|---|---|---|---|
| 3 | 0.4680 → 0.4597 | 63.57% → 67.96% | −0.3390 → −0.9145 |
| 5 | 0.8032 → 0.8115 | 37.15% → 39.28% | 1.5019 → 0.6822 |
The mixture improves probability scores and hit rates for both fish counts. Three-fish RMSE decreases, but five-fish RMSE increases. We also select the highest-weight component's mean as a single candidate, avoiding an average across components. This raises hit rates to 69.34% and 40.15%, with RMSEs of 0.4872 and 0.8716 L. This summary of the experiment has been updated in Section 5.1 (p. 8), the protocol and results in Supplement S5.6 (pp. 28–31, Tables S20–S21).
(b) We checked whether failed predictions lie near other fish or source-position averages.
For seven reference estimators on the additional test set, only 0.21–4.70% of failed predictions lie within 0.25 L of another actual source. Among the four neural models, the category near an equal-weight source-subset mean, while remaining far from every actual source, accounts for 3.97–4.98% of three-fish failures and 14.42–15.12% of five-fish failures. These percentages use failed predictions as the denominator.
Now these geometric averages are distinguished from conditional averaging over possible targets in different hidden scenes compatible with an observation. The spatial categories describe where errors occur. The explain has been added to Supplement S5.4 (p. 27, with Table S18 on p. 28).
(c) We tested whether reconstructing all sources improves nearest-target localization.
We trained joint source-set MLP and TCN models with known source count and all-source position supervision. On the original test set, two-source prediction improves hit rates but increases nearest-target RMSE. With four sources, both metrics worsen. This tests recovery of one complete source configuration, a different output task from proposing alternative locations for the nearest target. We report it in Supplement S3.5 (pp. 20–21, Table S10).
The multi-hypothesis comparison shows that changing the output formulation improves some prediction scores. However, far-from-all predictions still contribute 88.92% and 88.86% of the mixture mean's squared target-position error for three and five fish. Distinguishing ambiguity in the measured flow from limitations of the frozen representation or fitted head remains an open question. We discuss this interpretation in Supplement S5.6 (p. 31).
Thank you for suggesting this comparison. If we have misunderstood your concern or the intended multi-hypothesis test, we would appreciate your clarification so that we can address it during the rebuttal period and further improve the manuscript.
Update on Real-World Data Alignment
While you are reviewing our new updates according to your initial review, we are happy to share the good alignment between our benchmark and experimentally measured flow magnitudes. Thus, we believe these experiments significantly helped us improve our work and strengthen the experimental basis of our paper.
We examined both datasets. The Wolf and van Netten (2019) data contain sensor-deflection signals, so a verified calibration is needed for a direct velocity comparison. Thus, we compared the fish PIV dataset associated with Tack et al. (2024) with our flow generator with three recordings from three individual fish swimming at a nominal speed of 1 body length per second.
We first compared the mean flow fields and sampled corresponding locations on both sides of each fish. Across the three recordings, the mean absolute differences in streamwise and lateral flow velocity were 4.99% and 3.02% of the nominal swimming speed, respectively. We also compared temporal velocity fluctuations. The results showed that their spatially averaged RMS amplitudes differed from the PIV measurements by 1.81–6.15%, relative to the measured amplitudes. These results indicate good agreement between the synthetic and experimentally measured flow magnitudes at the sampled flank locations.
We have added these results in Section 3.1 in the main text and a new supplementary section, “Comparison with experimental flow measurements” (Supplement S8), including the details of the comparison procedure, flow-field plots, and quantitative tables.
Response to Review Comments 1/2
Thank you for your constructive feedback. These comments led us to make the physical evidence more explicit and to test the multi-source prediction approaches. We respond to the experimental-data concern first, then discuss what the new failure analysis shows. Page references are to the revised manuscript.
1. Connect the benchmark's physical foundation and scale to the challenges of obtaining experimental learning data.
To address this reasonable concern, we now add a clearer account of the published DNS evidence and the challenges of collecting experimental benchmark data.
The flow generator was built based on the hydrodynamic model of Zhou et al., Physics of Fluids (2025). That study used high-fidelity three-dimensional direct numerical simulations (DNS) to parameterize wake geometry and circulation decay. It also validated model-reconstructed vortex organization and time-averaged induced flow with DNS. We now add this inherited physical evidence in Section 3.1 (p. 3, bottom) and Supplement S1.1 (p. 13).
We also revised the Discussion to explain the practical gap between measuring a school and assembling a learning benchmark. Pan and Lauder (2024) demonstrate multi-fish particle image velocimetry (PIV), but report blocked illumination and camera occlusion within schools. These effects leave parts of the flow unmeasured and complicate synchronized flow and state recording. Building a typical sized benchmark also requires coverage across configurations with consistent labels and evaluation splits, which is extremely challenging with experimental swimmers and current techniques. We indeed plan to incorporate experimental measurements and independent DNS data in future work, as advances in PIV and lower simulation costs make systematically collected, well-labeled datasets more accessible.
Our common-prior task supplies 16,000 training scenes and separate validation and test sets (Table 1, p. 4). Because we keep each source's contribution, we can change interference while holding the geometry and target fixed. We demonstrate these controlled tests in Section 5.3 (pp. 9–10). The revised Discussion explains this contribution and identifies physical-sensor and free-swimming experiments as the next tests of transfer (p. 10). The published DNS evidence supports the inherited flow construction.
(Due to the length limit, continue on the next comment thread.)