- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses an important and valuable problem, building simulated fish grounded in real world shape and swimming patterns, that could be used to further explore animal behavior patterns and to create realistic controllable assets. The presented method Video2SwimFish aims to transform real fish videos into controllable assets with individual swimming policies. It defines a Biological Locomotion Manifold (BLM) of the fish, a PCA action space derived from each fish observed midline curvature. Instead of adjusting the joints directly, the policy adjusts the PCA coefficients. To train and evaluate the method, a new dual-view dataset is presented containing recordings of 120 fish from 6 species. The method is evaluated on 3 tasks: 1) whether reconstructed fish can follow real fish trajectories (12 fish) 2) whether observed swimming can be transferred to the reconstructed fish (6 fish), 3) whether the fish can serve as an interactive component of the environment for training downstream embodied agents (2 fish).
Critical strengths
The main strength of the paper is the synchronized top- and front-view video recordings of 6 fish species (overall 120 individual fish). Prior work such as 3D-ZeF (Pedersen et al., CVPR 2020) records only zebrafish. The variety of fish shape and swimming style makes those recordings valuable for study of individual morphology and locomotion, including fins movement, articulation and swimming policy which could be highly valuable for both animation and fish behavior research. The paper also presents a new pipeline for reconstruction of simulation-ready assets for each individual fish and the pipeline’s output on the collected videos. These strengths are, however, limited by the issues described under Weaknesses: the quality of the generated assets is validated on a very small set. Quantitative results cover 10% or less of the dataset (12/120 fish in Task 1, 6/120 in Task 2, 2/120 in the case study), and qualitative comparisons are limited to a few selected frames and videos.
Critical weaknesses
-
Dataset Annotations: Animal benchmarks usually provide some type of ground truth annotations, for example 3D-ZeF [1] provided ground-truth bounding boxes and head point annotations, and APT-36K [2] provides video keypoint annotations across different animals. A recent study on a similar species showed that fish keypoints can be annotated and accurately estimated in a similar top-view tank setting [3]. In contrast, the presented dataset is released without ground-truth annotations. The midline trajectories and body lengths used for evaluation of “real fish” are pseudo-labels from automatic segmentation without showing the accuracy or the visual quality of the labels and number of retained frames is not reported. An annotated subset is needed to make the dataset more valuable to future work and evaluation reliability.
-
Per individual fish results: the paper presents as the main contribution (L044, L103) 120 controllable fish assets and individual swimming policies derived from them, but reports only on 12,6, and 2 hand-selected fish (L341), and the authors state that “performance across all 120 assets therefore remains unestablished” (L476), against the main claim of the paper. Tables report only means, with no clear variance and visual comparisons between different individual fish. It is unclear whether the method and the assets work beyond these selected cases.
-
Realistic swimming comparison: While the paper states FishGym are “hand-designed rather than grounded in biological observation” (L144), the paper’s simulated also mentioned as “is not required to replicate the mechanical constraints of a real fish body” and uses joint and drive preset across all fish (L241). Since both approaches rely on non-biological joint models, the claimed advantage should be demonstrated, for example by comparing the generated fish with FishGym fish under the same fidelity metrics, or through a human study.
-
Benchmark and biological fidelity evaluation protocol: Biological fidelity is never formally defined and is measured against unvalidated pseudo-labels (curvature, frequency and speed). The BLM’s mixed results (Table 1) are hard to interpret as evidence of a tradeoff (L481), since they may reflect limitations of the method or of the evaluation. A synthetic test with known ground truth, such as rendering a FishGym fish with a known gait and running it through the full pipeline, would show whether the pipeline and the BLM recover real motion, and would make the fidelity claims convincing.
Minor weaknesses:
- L103 describes the output assets as a second dataset contribution, releasing the output assets supporting reproducibility and validation of the pipeline. To claim it’s part of the video dataset annotations all 120 assets should be verified and compared against high quality GT annotations.
- Prompt for the canonical frame and reconstructing it with a commercial platform api (Meshy) is a preprocessing step rather than a novel part of the pipeline. Evaluation compares 8/120 fish meshes on the same frame it was generated from. Comparisons against views not used for reconstruction, and on the full 120 fish is needed to understand the quality and limitations of the step.
- The BlueROV case has no baseline (a static or scripted target) with only 2 individual fish. The presented scenario seems to be too simple to show that the fish motion matters. In addition an explanation on when it fails and why would be useful to understand the limitation of the method.
[1] M. Pedersen, J. B. Haurum, S. H. Bengtson, T. B. Moeslund. 3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset. CVPR, 2020.
[2] Y. Yang, J. Yang, Y. Xu, J. Zhang, L. Lan, D. Tao. APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking. NeurIPS, 2022.
[3] S. K. Kumaran, L. E. Solberg, D. Izquierdo-Gomez, et al. Applying deep learning and the ecological home range concept to document the spatial distribution of Atlantic salmon parr (Salmo salar L.) in experimental tanks. Scientific Reports, 15:5976, 2025. https://doi.org/10.1038/s41598-025-90118-9
[4] W. Liu, K. Bai, X. He, S. Song, C. Zheng, and X. Liu. 2022. FishGym: A High-Performance Physics-based Simulation Framework for Underwater Robot Learning. ICRA 2022.
Questions for authors
-
Why is Qwen3-VL specifically needed to select the canonical frame?
-
For BLM, why exactly were 4 PCA components chosen? How do different numbers of components affect the results?
-
Actors use templates of 14 other species. Were the results verified against skeletons built for the 6 recorded species?
-
The actor is fine-tuned using 74 examples, (L230) How much does fine-tuning help compared with the base VLM (or a larger one)? Do the results improve if the critic model is larger than the actor one?
Three things that would help me shift toward acceptance: adding annotation to the dataset, expanding the pipeline validation to include all recoverable individual fish assets (120 or less and state the limitations), and evaluate the realism of the assets swimming policies both qualitatively and using a in human study against FishGym.
Response to Reviewer WwbE
We thank you for the detailed review and for recognizing the value of the dual-view recordings across six species. Please also see the Important Update on extending the BLM to 3D and the General Response.
Your three conditions for acceptance.
Dataset annotations: we manually annotated 432 synchronized frame pairs from 12 fish across all six species and release them with the dataset. We also used these annotations to validate our midline extraction pipeline: its midlines differ from the human labels by 0.8% of body length, close to the 0.7% difference between the two annotators, which shows that the pipeline is reliable (General Response, Point 2).
Validation of the pipeline: we clarify the role of the 120 assets and the scope of the evaluation (General Response, Point 4), and the mesh evaluation (General Response, Point 5).
Realism compared with FishGym: FishGym addresses a different problem: it provides hand-designed fish bodies for underwater robot learning, whereas we build controllable assets from videos of real fish. Our fidelity metrics compare each simulated fish with the real fish it was built from, so they are not defined for FishGym fish, which have no recorded counterpart. For perceived realism, we are contacting fish researchers for an expert study that compares our method with our baselines on the same assets. A synthetic recovery test with a known gait is left to future work.
Response to question: Why is Qwen3-VL needed to select the canonical frame? Single-image 3D reconstruction is sensitive to the pose in the input frame, so we need a frame in which the fish is fully visible, seen from the side, and close to its resting shape (Sec. 3.1). Simple rules based on silhouette area or aspect ratio cannot tell these cases apart from frames with a bent body, a partly hidden fish, or the fish's reflection in the tank wall, while a VLM can judge all of them at once.
Response to question: Why 4 PCA components? Please see General Response, Point 1.
Response to question: Templates from other species. The 14 expert templates intentionally come from species outside our recordings. Their length-to-height ratios (1.8 to 3.8) span the body shapes of the six recorded species, and using them tests the pipeline in its intended use: building assets for new species from new videos. Two expert models (roach and trevally) are held out to validate the actor during fine-tuning (App. G).
Response to question: Fine-tuning and model size. The critic is the base Qwen3-VL-32B model, the largest dense model in the Qwen3-VL family (Bai et al., 2025). A 32B open-weight model also keeps the pipeline runnable on local hardware, which matters for a toolkit released to the community. We will note testing a larger critic as future work.
Response to weakness: Dataset annotations. Please see condition (1) above and General Response, Point 2.
Response to weakness: Per individual fish results. We now report the standard deviation of every metric across fish (follow-up comment to the General Response) and explain the choice of the evaluated subset (General Response, Point 4). To compare individual fish visually, we release videos of the Task 1 rollouts at https://huggingface.co/datasets/video2swimfish/video2swimfish-dataset/tree/main/data/table1_results, organized by method, fish, and seed.
Response to weakness: Realistic swimming comparison. Please see condition (3) above.
Response to weakness: Benchmark and biological fidelity evaluation protocol. In General Response, Point 2, we define biological fidelity and validate the midline labels against manual annotations. The synthetic recovery test is discussed in condition (3) above.
Response to minor weakness: Assets as a second dataset contribution. Every asset is checked automatically for containment, coverage, and alignment (App. B.5), and was inspected by the authors before release. The manual midline annotations (General Response, Point 2) and the mesh evaluation (General Response, Point 5) provide ground-truth checks for the inputs and outputs of the pipeline.
Response to minor weakness: Meshy as a preprocessing step. We agree that Meshy is an off-the-shelf component, and we describe it as such (Sec. 3.1). Our contribution is the pipeline around it: selecting a canonical frame, scaling the mesh to the measured body length, rejecting and cleaning failed meshes (App. B.2), and building the articulation and the BLM, together with the paired datasets and benchmark. General Response, Point 5 discusses the mesh evaluation.
Response to minor weakness: BlueROV case study. The case study shows that the assets can be used as interactive prey for training an underwater robot, rather than serving as a benchmark task.
We hope these answers and the revision address your concerns, and thank you again for the careful review.