acceptodds
Withdrawn from ICLR 2027

VIDEO2SWIMFISH: AN AUTOMATED PIPELINE FOR RECONSTRUCTING CONTROLLABLE FISH MODELS AND BIOLOGICAL LOCOMOTION FROM REAL FISH VIDEOS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present **Video2SwimFish**, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor–critic loop, and extracts a Biological Locomotion Manifold (BLM) from the fish's observed midline curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from video, and a downstream case study in which a simulated BlueROV underwater robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Anonymous project website: https://video2swimfish.vercel.app.

Primary area:
datasets and benchmarks
Keywords:
Locomotion Learning, Reinforcement Learning, Simulation, Dataset, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WwbEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses an important and valuable problem, building simulated fish grounded in real world shape and swimming patterns, that could be used to further explore animal behavior patterns and to create realistic controllable assets. The presented method Video2SwimFish aims to transform real fish videos into controllable assets with individual swimming policies. It defines a Biological Locomotion Manifold (BLM) of the fish, a PCA action space derived from each fish observed midline curvature. Instead of adjusting the joints directly, the policy adjusts the PCA coefficients. To train and evaluate the method, a new dual-view dataset is presented containing recordings of 120 fish from 6 species. The method is evaluated on 3 tasks: 1) whether reconstructed fish can follow real fish trajectories (12 fish) 2) whether observed swimming can be transferred to the reconstructed fish (6 fish), 3) whether the fish can serve as an interactive component of the environment for training downstream embodied agents (2 fish).

Critical strengths

The main strength of the paper is the synchronized top- and front-view video recordings of 6 fish species (overall 120 individual fish). Prior work such as 3D-ZeF (Pedersen et al., CVPR 2020) records only zebrafish. The variety of fish shape and swimming style makes those recordings valuable for study of individual morphology and locomotion, including fins movement, articulation and swimming policy which could be highly valuable for both animation and fish behavior research. The paper also presents a new pipeline for reconstruction of simulation-ready assets for each individual fish and the pipeline’s output on the collected videos. These strengths are, however, limited by the issues described under Weaknesses: the quality of the generated assets is validated on a very small set. Quantitative results cover 10% or less of the dataset (12/120 fish in Task 1, 6/120 in Task 2, 2/120 in the case study), and qualitative comparisons are limited to a few selected frames and videos.

Critical weaknesses
  1. Dataset Annotations: Animal benchmarks usually provide some type of ground truth annotations, for example 3D-ZeF [1] provided ground-truth bounding boxes and head point annotations, and APT-36K [2] provides video keypoint annotations across different animals. A recent study on a similar species showed that fish keypoints can be annotated and accurately estimated in a similar top-view tank setting [3]. In contrast, the presented dataset is released without ground-truth annotations. The midline trajectories and body lengths used for evaluation of “real fish” are pseudo-labels from automatic segmentation without showing the accuracy or the visual quality of the labels and number of retained frames is not reported. An annotated subset is needed to make the dataset more valuable to future work and evaluation reliability.

  2. Per individual fish results: the paper presents as the main contribution (L044, L103) 120 controllable fish assets and individual swimming policies derived from them, but reports only on 12,6, and 2 hand-selected fish (L341), and the authors state that “performance across all 120 assets therefore remains unestablished” (L476), against the main claim of the paper. Tables report only means, with no clear variance and visual comparisons between different individual fish. It is unclear whether the method and the assets work beyond these selected cases.

  3. Realistic swimming comparison: While the paper states FishGym are “hand-designed rather than grounded in biological observation” (L144), the paper’s simulated also mentioned as “is not required to replicate the mechanical constraints of a real fish body” and uses joint and drive preset across all fish (L241). Since both approaches rely on non-biological joint models, the claimed advantage should be demonstrated, for example by comparing the generated fish with FishGym fish under the same fidelity metrics, or through a human study.

  4. Benchmark and biological fidelity evaluation protocol: Biological fidelity is never formally defined and is measured against unvalidated pseudo-labels (curvature, frequency and speed). The BLM’s mixed results (Table 1) are hard to interpret as evidence of a tradeoff (L481), since they may reflect limitations of the method or of the evaluation. A synthetic test with known ground truth, such as rendering a FishGym fish with a known gait and running it through the full pipeline, would show whether the pipeline and the BLM recover real motion, and would make the fidelity claims convincing.

Minor weaknesses:

  • L103 describes the output assets as a second dataset contribution, releasing the output assets supporting reproducibility and validation of the pipeline. To claim it’s part of the video dataset annotations all 120 assets should be verified and compared against high quality GT annotations.
  • Prompt for the canonical frame and reconstructing it with a commercial platform api (Meshy) is a preprocessing step rather than a novel part of the pipeline. Evaluation compares 8/120 fish meshes on the same frame it was generated from. Comparisons against views not used for reconstruction, and on the full 120 fish is needed to understand the quality and limitations of the step.
  • The BlueROV case has no baseline (a static or scripted target) with only 2 individual fish. The presented scenario seems to be too simple to show that the fish motion matters. In addition an explanation on when it fails and why would be useful to understand the limitation of the method.

[1] M. Pedersen, J. B. Haurum, S. H. Bengtson, T. B. Moeslund. 3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset. CVPR, 2020.

[2] Y. Yang, J. Yang, Y. Xu, J. Zhang, L. Lan, D. Tao. APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking. NeurIPS, 2022.

[3] S. K. Kumaran, L. E. Solberg, D. Izquierdo-Gomez, et al. Applying deep learning and the ecological home range concept to document the spatial distribution of Atlantic salmon parr (Salmo salar L.) in experimental tanks. Scientific Reports, 15:5976, 2025. https://doi.org/10.1038/s41598-025-90118-9

[4] W. Liu, K. Bai, X. He, S. Song, C. Zheng, and X. Liu. 2022. FishGym: A High-Performance Physics-based Simulation Framework for Underwater Robot Learning. ICRA 2022.

Questions for authors
  1. Why is Qwen3-VL specifically needed to select the canonical frame?

  2. For BLM, why exactly were 4 PCA components chosen? How do different numbers of components affect the results?

  3. Actors use templates of 14 other species. Were the results verified against skeletons built for the 6 recorded species?

  4. The actor is fine-tuned using 74 examples, (L230) How much does fine-tuning help compared with the base VLM (or a larger one)? Do the results improve if the critic model is larger than the actor one?

Three things that would help me shift toward acceptance: adding annotation to the dataset, expanding the pipeline validation to include all recoverable individual fish assets (120 or less and state the limitations), and evaluate the realism of the assets swimming policies both qualitatively and using a in human study against FishGym.

1 comment
AuthorsOpenReview ↗

Response to Reviewer WwbE

We thank you for the detailed review and for recognizing the value of the dual-view recordings across six species. Please also see the Important Update on extending the BLM to 3D and the General Response.

Your three conditions for acceptance.

  1. Dataset annotations: we manually annotated 432 synchronized frame pairs from 12 fish across all six species and release them with the dataset. We also used these annotations to validate our midline extraction pipeline: its midlines differ from the human labels by 0.8% of body length, close to the 0.7% difference between the two annotators, which shows that the pipeline is reliable (General Response, Point 2).

  2. Validation of the pipeline: we clarify the role of the 120 assets and the scope of the evaluation (General Response, Point 4), and the mesh evaluation (General Response, Point 5).

  3. Realism compared with FishGym: FishGym addresses a different problem: it provides hand-designed fish bodies for underwater robot learning, whereas we build controllable assets from videos of real fish. Our fidelity metrics compare each simulated fish with the real fish it was built from, so they are not defined for FishGym fish, which have no recorded counterpart. For perceived realism, we are contacting fish researchers for an expert study that compares our method with our baselines on the same assets. A synthetic recovery test with a known gait is left to future work.

Response to question: Why is Qwen3-VL needed to select the canonical frame? Single-image 3D reconstruction is sensitive to the pose in the input frame, so we need a frame in which the fish is fully visible, seen from the side, and close to its resting shape (Sec. 3.1). Simple rules based on silhouette area or aspect ratio cannot tell these cases apart from frames with a bent body, a partly hidden fish, or the fish's reflection in the tank wall, while a VLM can judge all of them at once.

Response to question: Why 4 PCA components? Please see General Response, Point 1.

Response to question: Templates from other species. The 14 expert templates intentionally come from species outside our recordings. Their length-to-height ratios (1.8 to 3.8) span the body shapes of the six recorded species, and using them tests the pipeline in its intended use: building assets for new species from new videos. Two expert models (roach and trevally) are held out to validate the actor during fine-tuning (App. G).

Response to question: Fine-tuning and model size. The critic is the base Qwen3-VL-32B model, the largest dense model in the Qwen3-VL family (Bai et al., 2025). A 32B open-weight model also keeps the pipeline runnable on local hardware, which matters for a toolkit released to the community. We will note testing a larger critic as future work.

Response to weakness: Dataset annotations. Please see condition (1) above and General Response, Point 2.

Response to weakness: Per individual fish results. We now report the standard deviation of every metric across fish (follow-up comment to the General Response) and explain the choice of the evaluated subset (General Response, Point 4). To compare individual fish visually, we release videos of the Task 1 rollouts at https://huggingface.co/datasets/video2swimfish/video2swimfish-dataset/tree/main/data/table1_results, organized by method, fish, and seed.

Response to weakness: Realistic swimming comparison. Please see condition (3) above.

Response to weakness: Benchmark and biological fidelity evaluation protocol. In General Response, Point 2, we define biological fidelity and validate the midline labels against manual annotations. The synthetic recovery test is discussed in condition (3) above.

Response to minor weakness: Assets as a second dataset contribution. Every asset is checked automatically for containment, coverage, and alignment (App. B.5), and was inspected by the authors before release. The manual midline annotations (General Response, Point 2) and the mesh evaluation (General Response, Point 5) provide ground-truth checks for the inputs and outputs of the pipeline.

Response to minor weakness: Meshy as a preprocessing step. We agree that Meshy is an off-the-shelf component, and we describe it as such (Sec. 3.1). Our contribution is the pipeline around it: selecting a canonical frame, scaling the mesh to the measured body length, rejecting and cleaning failed meshes (App. B.2), and building the articulation and the BLM, together with the paired datasets and benchmark. General Response, Point 5 discusses the mesh evaluation.

Response to minor weakness: BlueROV case study. The case study shows that the assets can be used as interactive prey for training an underwater robot, rather than serving as a benchmark task.

We hope these answers and the revision address your concerns, and thank you again for the careful review.

Reviewer_oGLpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Video2SwimFish constructs controllable fish assets from synchronized top and front videos. The pipeline combines mesh generation from images, articulation construction with a VLM, and a Biological Locomotion Manifold obtained by applying PCA to observed body curvature. The paper contributes recordings and derived assets for 120 fish across six species. It evaluates trajectory following, imitation from video, and a simulated robot capture task, measuring both task performance and agreement with observed locomotion.

Critical strengths
  1. The paired resource and complete pipeline are useful contributions. Each simulated asset is linked to the individual fish from which it was constructed. This correspondence provides a concrete reference for evaluating locomotion fidelity. The collection spans six species, and the experiments demonstrate that the resulting deformable, articulated assets support both locomotion learning and interaction with a simulated robot (Secs. 3–4). The integration has practical value for underwater embodied learning.

  2. The benchmark reveals meaningful differences between task performance and locomotion fidelity. In Table 1, BLM achieves the highest completion rate, 0.89, while Joint RL + AMP produces smaller trajectory and curvature errors but completes only 0.51 of episodes. Table 2 also shows that better curvature and speed matching does not imply better frequency matching. Evaluating these quantities together makes the benchmark informative beyond a single task-success measure.

Critical weaknesses
  1. The benefits of the proposed components are not sufficiently isolated. BLM combines PCA with empirical coefficient and rate bounds (Sec. 3.3), but there is no comparison between individual and species-level representations. The articulation stage also lacks a geometric baseline. This matters because deterministic completion expands roughly 5–6 actor-proposed bones into 12–19 bones (Appendix B.4). The current experiments leave unclear how much benefit comes from individual modeling and VLM refinement.

  2. Validation covers a selected subset of the collection. Task 1 uses 12 fish and Task 2 uses six, selected from valid assets with the cleanest videos (Sec. 4.1; Appendix E.4). Reconstruction fidelity is measured on eight fish against their canonical input images (Appendix C). These results support the demonstrated cases, but provide limited evidence of fidelity and simulation robustness across all 120 assets. Tables 1–2 also report no variability across fish or random seeds.

Questions for authors
  1. Evaluation clarification: How are video windows separated between BLM fitting, policy training, and final evaluation? Section 4.3 mentions a held-out validation split, but it is unclear whether the final evaluation uses temporally held-out clips (Appendices B.7 and E.2).

  2. Additional evidence: Are results available for a species-pooled BLM with the same dimensionality and for a geometric skeleton baseline using the same structural completion procedure? These comparisons would clarify the benefits of individual modeling and the VLM construction stage.

1 comment
AuthorsOpenReview ↗

Response to Reviewer oGLp

We thank you for the positive assessment, and for recognizing the value of the paired resource and of evaluating task success alongside locomotion fidelity. Please also see the separate Important Update post on extending the BLM from 2D to 3D.

Response to question: How are video windows separated between BLM fitting, policy training, and final evaluation? The final evaluation uses video clips that are never used in training: they are separate from the clips used for policy training, BLM fitting, and the reference data of every baseline (General Response, Point 3).

Response to question: Species-pooled BLM and geometric skeleton baseline. We will compare each fish's own BLM with a species-pooled BLM with the same number of modes (General Response, Point 1). For the articulation, as explained in General Response, Point 6, a uniform chain of equally spaced, identical bones ignores the shape of the fish, whereas the VLM actor sets the position and size of each bone to fit the local body width. Point 6 also describes how we quantify the contribution of the VLM stage.

Response to weakness: The benefits of the proposed components are not sufficiently isolated. Please see our answer to the second question above (General Response, Points 1 and 6).

Response to weakness: Validation covers a selected subset of the collection. The 120 fish are released as a dataset and toolkit for the community, and we evaluate on a fixed subset that covers all six species, following common practice for large benchmarks (General Response, Point 4). Task 1 now reports results over 3 seeds per method, with standard deviations of every metric across fish (General Response, Point 3, and the follow-up comment); this variability comes from differences between fish, while results for the same fish are stable across seeds. The mesh IoU is reported only on cleanly segmented fish because, for the others, it would mainly measure segmentation errors (General Response, Point 5).

We hope these updates address your concerns, and thank you again for your review.

Reviewer_mGU4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a controllable simulated fish from synchronized top and front video. Qwen3-VL picks a canonical frame, Meshy makes a scaled mesh, and a VLM actor-critic loop places bones before a deterministic completion step. The action space is the BLM: a per-fish PCA of midline curvature (four modes, 99.92% of variance, rate limits from the 99th percentile), with a pooled basis for a few fish. The authors release 120 fish from six species, 20 each, plus the assets. Benchmarks are trajectory following on 12 fish, reward-free cloning from video on 6, and an ROV capture case study with 2. The main finding is that task success and locomotion fidelity don't improve together.

Critical strengths
  1. The dataset is the strongest part. 120 identified fish, six species, two synchronized cameras at 32 fps and 2736x1824, ChArUco calibration, IACUC-approved protocol (Sec. 3.4, App. A). One fish per recording, which is a real difference from 3D-ZeF as the paper describes it. I would use the data without the pipeline.

  2. The paper reports task success next to kinematic fidelity, and it doesn't hide the negative result. In Table 1, BLM+RL has the best completion (0.89) and the best frequency and speed errors, but the worst Fréchet distance (0.65 BL). Joint RL+AMP has the best path (0.34 BL) and curvature W1 (0.78) but completes 0.51. Baselines share fish, actuators, PPO settings and reference windows, and CPG parameters are fitted per fish. The paper says no method wins on everything.

Critical weaknesses
  1. The VLM articulation step is one of the main claimed contributions, but nothing shows what it adds. There is no ground truth or human rating for the skeletons; the fourteen hand-built models are fine-tuning data and templates, not an evaluation. Beyond the geometric checks (87/120 pass strict containment, App. B.5), the main quality signal is the base VLM critic, which is given measured midline facts, re-scores after the completion step, and gives 114/120 fish a 5 (App. B.4). The actor usually places 5-6 bones; the exported chains have 12-19 (median 15). The verifier snaps bones to the centerline and all fish share one joint preset (App. G), so most of the final skeleton is deterministic geometry. Without a comparison to a uniform centerline chain plus completion, a non-fine-tuned actor, or an existing auto-rigging method, I can't tell what the VLM contributes.

  2. The benchmark has no uncertainty, and part of it is scored during training. Tables 1 and 2 are means over 12 and 6 fish with no standard deviations, and I can't find the number of seeds. Task 1 metrics are "logged every epoch on 16 tracked environments" (App. E.1), and the case-study success is the training-time "mean of the last outcome of every environment" (App. E.3), not a frozen-policy evaluation. Runs stop once completion stays at or above 0.90 for three epochs, or plateaus, so completion also drives the stopping rule, and training length per method isn't reported. Fréchet distance leaves out blow-ups, but Table 1 has no blow-up rates, so methods may be compared on different episode subsets. The evaluated fish are the ones "with the most clean video frames" (App. E.4). I don't think later work could use this as a reference point yet.

Questions for authors
  1. Are Tables 1 and 3 from training logs or frozen-policy evaluation? How many seeds and epochs per method, what are the standard deviations across fish and seeds, and what are the blow-up rates?
  2. Can you run Task 1 with a non-VLM skeleton (uniform centerline chain plus completion)? A critic score distribution before completion, and the share of bones the actor actually places, would also help.
  3. The "own PCA basis" column in Table 4 sums to 108/120, but App. B.7 says 7/120 use a pooled basis. Which is right, and are any pooled-basis fish in Tasks 1 or 2?
  4. The 0.88 IoU (Sec. 3.1) covers 8 fish from 4 species, the two cleanest segmentations per species, measured against the generating frame after an IoU-maximizing affine fit (App. C). Can you report it for all 120, or on a held-out view?
  5. How well does the curvature-to-joint controller track the commanded curvature? BLM+RL has worse curvature W1 than Joint RL+AMP (1.01 vs 0.78).

Other suggestions:

  • Ablations of the BLM rate bound, the number of modes, and individual vs pooled bases.
  • The panel model leaves out added mass (App. G.1), so I'd scope "biological fidelity" to midline kinematics under this surrogate, and relate it to Lighthill (1971).
  • Swim2Real (arXiv 2603.20827, 2026) calibrates a robotic fish simulator from video. Here actuator and hydrodynamic settings are fixed by hand, so calibrating them from the recorded videos seems a natural extension.
  • Possibly relevant related work: RigNet (Xu et al., 2020), Stephens et al. (2008) on the "eigenworm" posture description, and SoftCon (Min et al., 2019).
  • Heading stability is tied in Table 2 (0.56 vs 0.56), so "ahead on every fidelity metric except tail-beat frequency" (Sec. 4.3) should be softened.
  • Please confirm that Meshy's terms allow redistributing the meshes under CC BY-NC 4.0, and add the IACUC protocol number in the camera-ready.
5 comments
AuthorsOpenReview ↗

Response to Reviewer mGU4

We thank you for the detailed and constructive review, and for recognizing the value of the dataset and of reporting task success alongside kinematic fidelity. Please also see the separate Important Update post on extending the BLM from 2D to 3D.

Response to question: Are Tables 1 and 3 from training logs or frozen-policy evaluation? In the submitted version, the Task 1 metrics were logged during training on 16 tracked environments (App. E.1), and the case-study success rate was the training-time mean described in App. E.3. In the revision, all reported metrics are computed by frozen-policy evaluation on held-out clips, with 3 seeds per method, blow-up rates, and the training budget and stopping rule of each method (General Response, Point 3). The standard deviations of all metrics are given in the follow-up comment to the General Response.

Response to question: Task 1 with a non-VLM skeleton, the critic score distribution before completion, and the share of bones placed by the actor. Please see General Response, Point 6.

Response to question: Table 4 vs. App. B.7. App. B.7 is correct: 7 of the 120 fish use a species-pooled basis, and we will correct Table 4 accordingly. None of the fish in Tasks 1 and 2 use a pooled basis.

Response to question: Mesh IoU for all 120 fish or on a held-out view. Please see General Response, Point 5.

Response to question: How well does the curvature-to-joint controller track the commanded curvature? Very accurately. Replaying real BLM coefficients open-loop, the controller reproduces the commanded per-joint curvature with R² = 0.992 ± 0.005 (lateral) and 0.993 ± 0.001 (vertical) across the 12 fish (RMSE 0.063 and 0.056 κ·BL, about 8.5% of the standard deviation of the command), with no measurable lag at 30 Hz and a midline error of 0.007 BL. The higher curvature W₁ of BLM + RL therefore does not come from tracking error; it reflects the curvature patterns that the policy selects to complete the task, consistent with the trade-off between task success and locomotion fidelity.

Response to weakness: The VLM articulation step is one of the main claimed contributions, but nothing shows what it adds. Please see General Response, Point 6. Regarding existing auto-rigging methods, RigNet and similar methods predict animation skeletons and skinning weights for the surface meshes of characters, whereas our assets need a chain of rigid internal bones that stays inside the skin and drives an FEM body.

Response to weakness: The benchmark has no uncertainty, and part of it is scored during training. Please see our answer to the first question above. General Response, Point 4 explains how the evaluated fish are selected.

Response to suggestion: Ablations of the BLM rate bound, the number of modes, and individual vs. pooled bases. General Response, Point 1 discusses the number of modes and the individual vs. pooled comparison. The rate bound is the 99th percentile of the per-step coefficient change in each fish's own video, so it leaves the policy the full rate of change seen in 99% of real-fish steps.

Response to suggestion: Scope of biological fidelity and added mass. Please see General Response, Point 2.

Response to suggestion: Calibrating actuator and hydrodynamic settings from video. We agree that this is a natural extension, as in Swim2Real, and we will discuss it as future work.

Response to suggestion: Related work. We will add RigNet (Xu et al., 2020) and SoftCon (Min et al., 2019), and discuss Stephens et al. (2008): the BLM follows the same idea as the eigenworm posture description, applied to each individual fish and used as an action space.

Response to suggestion: Sec. 4.3 wording. We agree that heading stability is tied in Table 2, and we will revise the sentence so that it reports ties accurately.

Response to suggestion: Meshy license and IACUC protocol number. The meshes were generated through Meshy's paid API, and Meshy's terms grant full ownership of assets generated on paid plans, which permits redistribution under CC BY-NC 4.0. The IACUC protocol number (LA26-0016) is given in the Ethics Statement.

We hope these answers and the revision address your concerns, and thank you again for the careful review.

References

  • Min, S., et al. (2019). SoftCon: Simulation and Control of Soft-Bodied Animals with Biomimetic Actuators. ACM Transactions on Graphics (SIGGRAPH Asia 2019).
  • Stephens, G. J., Johnson-Kerner, B., Bialek, W., & Ryu, W. S. (2008). Dimensionality and dynamics in the behavior of C. elegans. PLoS Computational Biology, 4(4), e1000028.
  • Swim2Real. arXiv:2603.20827 (2026).
  • Xu, Z., et al. (2020). RigNet: Neural Rigging for Articulated Characters. ACM Transactions on Graphics (SIGGRAPH 2020).
Reviewer_mGU4OpenReview ↗

response

Thank you for the detailed response and the new experiments. The frozen-policy evaluation on held-out clips, the three seeds, the stated stopping rule and training budgets, and the standard deviations for every metric address my main concern about benchmark uncertainty. The midline annotations (0.8% BL against 0.7% BL between annotators), the controller tracking result, and the Table 4 correction are also helpful. Some points remain.

  1. The revised Table 1 changes the results, and the General Response describes the change inaccurately. Point 3 says "As in the submission, BLM + RL achieves the highest completion and the lowest Fréchet distance". In the submission, BLM + RL had the highest Fréchet distance (0.65 BL, worst of five) and Joint RL + AMP the lowest (0.34 BL). In the revision this is reversed: BLM + RL is lowest (0.67) and Joint RL + AMP highest (0.78). BLM + RL also moves from the lowest Δf (0.56) to the second highest (1.29), and its W₁(κ) is now the worst in the table (1.13). Its completion lead shrinks from 0.89 against 0.70 to 0.70 against 0.68 (CPG + RL), and CPG + RL converges about as fast (112 ± 41 against 119 ± 42 epochs). The 2D to 3D change, the new stopping rule and the new evaluation were introduced together, so I can't tell which of them causes these shifts. A trade-off between completion and fidelity is still visible, but which method sits where on it depends on the protocol, and the Sec. 4.2 text ("BLM+RL completes 0.89 ... but the largest Fréchet distance", "Joint RL+AMP is its mirror image") no longer matches the table.

Since the controller tracks commands with R² = 0.99, the worst W₁(κ) for BLM + RL means the policy picks curvature patterns that are far from the real fish while staying inside a manifold fitted to that fish. This weakens the argument that bounding the action space by real-fish motion helps fidelity, and I think the paper should say so.

Could you report a paired per-fish comparison of BLM + RL and CPG + RL (e.g. on how many of the 12 fish BLM + RL is better by more than the seed-to-seed spread), and revise the claims about the BLM's advantage in completion and convergence accordingly?

  1. Tasks 2 and 3. The new protocol is shown only for Task 1. Are Table 2 (imitation) and Table 3 (case study) also evaluated with frozen policies, held-out clips and several seeds?

  2. VLM articulation. I don't think a geometric baseline is uninformative. It doesn't need identical bones: a chain along the centerline with each bone sized from the local body width, followed by the same completion step and containment check, is easy to build and is exactly the alternative the VLM has to beat. The argument that bone size should follow local body width describes what such a rule would also do. The comparison with a non-fine-tuned actor is also not addressed. The critic score distribution before completion and the share of actor-placed bones were promised; could you point me to them in the revision? Without some comparison, the articulation step remains a claimed contribution without evidence of what it adds.

  3. Mesh IoU on a held-out view. I accept the segmentation argument for the remaining fish, but the held-out view question is still open. The front camera is synchronized with the top view, so for the same eight fish the mesh silhouette could be compared with the front view, or with another top-view frame, which would separate reconstruction quality from fitting the generating frame.

Minor. Please report the number of 3D BLM modes and the variance they explain. Is the promised own vs. species-pooled BLM comparison in the revision?

The uncertainty fixes are a real improvement, but they also show that the BLM's advantage over CPG + RL is within noise, and the VLM contribution is still not quantified. For now I keep my score. I would reconsider if the VLM step were compared against a width-adaptive geometric skeleton (or the promised critic and bone-share numbers were convincing) and Sec. 4 were rewritten to match the new results.

Thank you for the careful reading and for catching this. Our description in General Response, Point 3 was wrong: in the submission, BLM + RL had the highest Fréchet distance, not the lowest. We have corrected it in the General Response.

On W₁(κ) and the fidelity argument for the BLM. We agree that staying inside the manifold does not make the policy reproduce the real fish's distribution of body shapes, and we will state this in the paper. However, we do not think the two distributions should match exactly. The recordings show what each fish did during a 2.5-minute session in our tank, not every motion it can make. Moreover, the simulated fish is asked to follow specific paths, so it uses its repertoire in different proportions than the freely swimming real fish. A distribution distance such as W₁(κ) penalizes both effects, even when the simulated fish moves entirely within the real fish's range. What we can require is that the simulated fish reproduces the motions the real fish showed. This is what motion coverage measures (the recall of Kynkäänniemi et al., 2019), and it is why the revised paper uses motion coverage and bending-envelope correlation instead of W₁(κ) and Δf. BLM + RL covers 0.85 of the real motions, against at most 0.78 for any baseline. For completeness, we will keep W₁(κ) and Δf in the appendix.

We will reply to your other points shortly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/2)

We thank all reviewers for their constructive reviews, and are encouraged that they found the dual-view dataset valuable (mGU4, oGLp, WwbE) and appreciated that the benchmark reports task success alongside locomotion fidelity (mGU4, oGLp). Reviewer-specific questions are answered in individual responses.

1. Response to question: Number of modes and individual vs. species-pooled BLM (mGU4, oGLp, WwbE)

  • Number of modes: the BLM is a principal component analysis (PCA) of each fish's body curvature, and we keep the smallest number of modes that together explain more than 98% of the variance. For the original BLM this gives 4 modes (99.92% of the variance); more modes add almost no new body shapes but make exploration noisier. We use the same rule for the 3D BLM.
  • Own BLM vs. species-pooled BLM: we will compare each fish's own BLM with a species-pooled BLM with the same number of modes, by their reconstruction error on held-out video frames.

2. Response to question: Unvalidated midline pseudo-labels and the definition of biological fidelity (mGU4, WwbE)

  • Accuracy of the midlines: to validate the automatically extracted midlines, two annotators manually labeled 432 synchronized top/front frame pairs from 12 fish (two per species, 36 frames per fish, sampled to cover a wide range of poses). Our automatic midlines are almost as accurate as a human annotator: their Chamfer distance to the human labels is 0.8% of body length (BL), close to the 0.7% BL between the two annotators. We release these annotations with the dataset and describe the protocol in the supplementary material.
  • Meaning of biological fidelity: we define biological fidelity as how closely the body kinematics of the simulated fish (body curvature, tail-beat frequency, and swimming speed) match those of the recorded individual, under our hydrodynamic model. This simplified panel model computes the drag on each surface patch from its velocity and omits added mass (the extra force from accelerating the surrounding water); full fluid–structure simulation would not allow the 256 parallel environments per GPU that our training uses (App. D). Since every method and every fish share the same model, differences between methods reflect the controllers rather than the fluid model (App. G.1). We will state this scope in the limitations and relate it to Lighthill (1971).

3. Response to question: Benchmark uncertainty, frozen-policy evaluation, and separation of video windows (mGU4, oGLp, WwbE)

In the revised benchmark:

  • We have rerun Task 1 (Table 1) on the same 12 fish with 3 seeds per method (180 runs). The revised Table 1 reports held-out test results (mean over 12 fish; ± std for completion). As in the submission, BLM + RL has the highest completion; unlike in the submission, it now also has the lowest Fréchet distance (previously the highest), while Joint RL + AMP still matches body curvature best, so a trade-off remains. We attribute the Fréchet change mainly to the 3D task: climbing and diving paths need coordinated lateral and vertical bending, which each BLM mode couples as in the real fish, while joint-space methods must discover it (Figure 3). | Method | Completion ↑ | Fréchet ↓ | W₁(κ) ↓ | Δf ↓ | Δv ↓ | |---|---|---|---|---|---| | Joint RL | 0.50 ± 0.29 | 0.76 | 0.88 | 0.92 | 0.23 | | Joint RL + AMP | 0.46 ± 0.28 | 0.78 | 0.81 | 0.80 | 0.24 | | CPG + RL | 0.68 ± 0.29 | 0.69 | 0.88 | 1.61 | 0.17 | | BCO + RL | 0.52 ± 0.29 | 0.75 | 0.90 | 0.93 | 0.21 | | BLM + RL (ours) | 0.70 ± 0.28 | 0.67 | 1.13 | 1.29 | 0.18 |

  • Blow-up rate, training budget, and stopping rule: none of the 36 runs per method became numerically unstable (blew up), so all metrics, including Fréchet distance, use every test episode. All methods share one stopping rule: training stops when the smoothed completion stays ≥ 0.95 for 10 epochs (convergence), or, after at least 150 epochs, when it improves by ≤ 0.02 over 100 epochs (plateau). One epoch is 65,536 environment steps; the 800-epoch cap was never reached. We evaluate the checkpoint with the highest training completion. BLM + RL and CPG + RL converge in about half of their runs and need fewer steps, while the other methods almost always plateau: | Method | Epochs | Env. steps (M) | Stopping reason | |---|---|---|---| | BLM + RL (ours) | 119 ± 42 | 7.8 ± 2.8 | converged 16, plateau 20 | | CPG + RL | 112 ± 41 | 7.2 ± 2.7 | converged 18, plateau 18 | | Joint RL | 148 ± 13 | 9.6 ± 0.9 | converged 1, plateau 35 | | Joint RL + AMP | 150 ± 0 | 9.8 ± 0.0 | plateau 36 | | BCO + RL | 147 ± 14 | 9.5 ± 0.9 | converged 3, plateau 33 |

  • All results come from frozen-policy evaluation (no further policy updates) on test clips that are held out from all training: they are not used to train the policies, to fit the BLM, or to build the reference data of any baseline.

AuthorsOpenReview ↗

Follow-up: Revised Paper Uploaded and Kind Reminder

Dear Reviewer,

Thank you again for your valuable feedback. We would like to kindly follow up and let you know that we have uploaded a revised version of the paper, which incorporates the changes described in our response.

We would greatly appreciate it if you could let us know whether the revision and our response have adequately addressed your concerns. If so, we would be grateful if you could consider updating your score by Nov 17 12:00 UTC. Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.