- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GenTri, an inference-time framework for occlusion-robust multi-view 3D pose estimation that combines a frozen, unconditional flow-matching pose prior with calibrated multi-view geometry. The method introduces an adaptive Cauchy reprojection objective to suppress unreliable 2D detections and a damped Gauss–Newton correction at the predicted clean endpoint to improve guidance efficiency without backpropagating through the flow network. An identity approximation of the endpoint Jacobian enables efficient per-joint optimization, while multiple generated hypotheses are selected according to their final robust reprojection energy. Experiments across three mouse domains, multiple 2D detection settings, and Human3.6M demonstrate substantial improvements under synthetic occlusion.
Critical strengths
-
Clear technical motivation. The paper identifies two important limitations of existing observation-guided inference methods: sensitivity to incorrect 2D detections under L2 reprojection and inefficient first-order corrections that do not exploit the directional constraints of multi-view geometry.
-
Well-controlled experimental comparisons. The authors adapt several existing guidance methods to the same pose prior and multi-view observation model, allowing a relatively fair assessment of the proposed guidance mechanism.
-
Evaluation across multiple domains and detectors. Experiments cover three mouse domains, including out-of-domain recordings and a newly collected treadmill dataset, as well as transfer to human pose estimation without retuning inference hyperparameters.
Critical weaknesses
-
Limited real-world occlusion validation. The occlusion benchmarks rely on synthetic occluders, and the paper does not establish whether the reported improvements generalize to naturally occurring occlusions in real experimental environments.
-
Insufficient analysis of hypothesis selection. Although Figure 8 compares different hypothesis aggregation strategies, the paper does not quantify the gap between lowest-energy selection and oracle selection based on ground-truth 3D error.
-
Dependence on prior quality. The method requires a separately trained species-specific flow prior, but its sensitivity to prior quality, training-data availability, and transfer to previously unseen species remains unexplored.
-
Limited analysis of multi-view robustness. The paper does not systematically evaluate how performance changes with fewer available cameras or increasing numbers of simultaneously corrupted views, despite relying on cross-view consistency to reject incorrect observations.
-
Insufficient detector diversity on Human3.6M. The human experiments rely exclusively on ResNet-152-based detectors, without evaluating whether the proposed guidance generalizes to other commonly used 2D pose estimators such as HRNet or CPN.
-
Incomplete validation of the Jacobian approximation. Although Figure 5(b) demonstrates the efficiency advantages of avoiding the full endpoint Jacobian, the approximation-induced accuracy difference is not explicitly isolated under identical sampling configurations.
-
Unexplained differences across species. The improvement over strong competing methods is considerably smaller on Human3.6M-Occluded than in several mouse settings, without sufficient analysis of whether this difference arises from the occlusion protocol, prior quality, or skeletal structure.
Questions for authors
-
Real-world occlusion: Can the authors provide results on naturally occurring occlusions, or quantitatively compare the synthetic occlusion patterns with those observed in real animal experiments?
-
Hypothesis selection: How reliable is the best-of-N selection strategy under severe occlusion, and what is the gap between the selected hypothesis and the oracle hypothesis with the lowest ground-truth 3D error?
-
Prior quality: How does GenTri perform when the flow prior is trained using substantially less 3D data, or when the prior is transferred to a new species with limited annotations?
-
Multi-view robustness: How does reconstruction accuracy change when the number of available camera views decreases or when multiple views contain confidently incorrect detections simultaneously?
-
2D detector generalization: Why is ResNet-152 exclusively adopted for Human3.6M, while DINOv2 is used for mouse experiments, and can the authors evaluate GenTri with HRNet or CPN to demonstrate robustness across different 2D detection error distributions?
-
Jacobian approximation: Can the authors report the accuracy difference between the identity-Jacobian approximation and the full endpoint Jacobian using identical guidance steps and hypotheses, in addition to the existing accuracy–latency comparison?
-
Cross-species performance: Why are the improvements over strong competing methods smaller on Human3.6M-Occluded than in several mouse experiments, and can the authors disentangle the effects of occlusion severity, skeletal structure, and prior quality?
6 comments
Response to Reviewer oHvg (Part 2/3)
[oHvg] asked how GenTri depends on prior quality, training-data size and transfer to a new species.
Thank you for this question. To test how GenTri depends on its prior, we ran three experiments that vary the amount of prior data, the species, and the quality of the training labels.
(i) Amount of training data. We trained priors on 50%, 25% and 10% of the training poses (Table R16). Even with 10%, the error rises by only 0.5 mm on mouse and 2.0 mm on human, so GenTri needs far less prior data than we used.
Table R16. Prior training size. Overall 3D error (mm) on Mouse-Occluded and Human3.6M-Occluded.
| Training data | Mouse | Human |
|---|---|---|
| 100% | 10.2 | 30.7 |
| 50% | 10.4 | 30.7 |
| 25% | 10.4 | 30.9 |
| 10% | 10.7 | 32.7 |
(ii) Transfer to a new species. Because keypoint definitions differ across species, a prior is trained per species, and what transfers is the procedure for building it. On Rat7M [1], a prior learned from 106 poses pseudo-labelled by robust triangulation, without 3D GT, lowers the error from 15.2 to 13.3 mm, close to 12.8 mm with a motion-capture prior (Table R17). A useful prior for a new species can thus be built from a small number of unannotated recordings.
Table R17. Prior training on Rat7M [1]. Pseudo-labels are obtained by prior-free robust triangulation.
| Prior training data | Poses | MPJPE (mm) |
|---|---|---|
| Without prior | – | 15.2 |
| Pseudo-labels | 106 | 13.3 |
| 3D GT | 106 | 12.8 |
(iii) Quality of the training labels. We trained mouse priors on 99 pseudo-labels triangulated from occluded treadmill images, on 99 from clean images, and on 99 GT poses (Table R18). The occluded-image prior is within 0.2 mm of the clean-image prior on treadmill and nearly matches the others on open-field, so noisy labels from occluded images are sufficient.
Table R18. Prior training with occluded observations. MPJPE (mm) on Mouse-Occluded. †Not evaluated: these poses were used to train the GT prior.
| Open-field | Open-field | Treadmill | |
|---|---|---|---|
| Prior training data | InD | OoD | OoD |
| 3D GT, treadmill (99 poses) | 5.0 | 15.2 | –† |
| Pseudo-labels, clean images | 5.2 | 15.2 | 9.4 |
| Pseudo-labels, occluded images | 5.4 | 15.3 | 9.6 |
Overall, GenTri is robust to both the amount and the quality of prior data, and its prior can be built for a new species without 3D annotation.
Revision: Section 4.3 (Table 6) and Appendix B.4 (Tables 22 and 24).
[oHvg] asked how accuracy changes with fewer cameras and when several views are confidently wrong.
Thank you for this question. To examine both cases, we evaluated GenTri and RANSAC on randomly reduced camera subsets, grouped joints by the number of correct views, and compared low-confidence with confidently wrong detections. GenTri is more accurate than RANSAC at every camera count, and the margin widens as cameras are removed: with two cameras it halves RANSAC's error on in-domain open-field and treadmill (Table R19). When several views are wrong at once, the gain is likewise largest where few correct views remain: with only 0 to 2 correct views, GenTri lowers RANSAC's error by a quarter to over a third (Table 12).
Table R19. Fewer cameras. Absolute MPJPE (mm) on Mouse-Occluded.
| Open-field InD | Open-field OoD | Treadmill OoD | ||||
|---|---|---|---|---|---|---|
| Cameras | RANSAC | Ours | RANSAC | Ours | RANSAC | Ours |
| 2 | 26.5 | 13.3 | 136.6 | 35.9 | 41.9 | 20.6 |
| 3 | 17.6 | 8.4 | 266.9 | 54.0 | 23.0 | 12.0 |
| 4 | 8.8 | 6.5 | 37.6 | 36.2 | 14.0 | 9.5 |
| 5 | 7.2 | 5.7 | 24.2 | 23.4 | – | – |
| 6 | 6.4 | 5.1 | 20.4 | 16.1 | – | – |
Detector confidence alone cannot handle confidently wrong detections, which we define as wrong detections (2D error above 10 mm) whose confidence is at least the median confidence of correct ones. Confidence weighting leaves these joints at 47.2 mm, whereas GenTri's Cauchy weighting, which uses reprojection residuals rather than confidence, reduces them to 18.3 mm, and the whole-pose prior to 15.0 mm (Table R20). GenTri's robustness therefore comes from geometric consistency across views, not from the detector's confidence, which is why it holds when the detector is confidently wrong.
Table R20. Wrong detections by confidence. Mean 3D error (mm).
| Wrong detections | Confidence | + Cauchy | + Prior (Ours) |
|---|---|---|---|
| Low-confidence wrong | 10.0 | 5.6 | 5.3 |
| Confidently wrong | 47.2 | 18.3 | 15.0 |
Revision: Appendix A.4 (Tables 9 and 10).
Response to Reviewer oHvg (Part 3/3)
[oHvg] asked why ResNet-152 is used for Human3.6M and whether the gains hold with other detectors such as HRNet.
Thank you for this suggestion. To show that GenTri's gains do not depend on one detector, we replaced ResNet-152 with HRNet-W48 on Human3.6M-Occluded, keeping GenTri's settings unchanged. HRNet's 2D errors are almost three times larger, which raises the error of every method, but GenTri remains the most accurate and its margin grows: it nearly halves RANSAC's error and improves on NRDF-MV by over a fifth (Table R21). GenTri therefore generalizes across detectors and their error distributions, and helps most when the detector is worse. For mouse, we use DINOv2, a pretrained foundation model, for its transferable features and strong out-of-domain detection with limited mouse training data.
Table R21. ResNet-152 vs. HRNet-W48 on Human3.6M-Occluded. Absolute MPJPE (mm) ↓.
| 2D detector | ResNet-152 | HRNet-W48 |
|---|---|---|
| Mean 2D error (px) | 9.5 | 26.1 |
| 2D error >50 px | 3.5% | 15.3% |
| Triangulation | 57.0 | 354.7 |
| RANSAC | 38.8 | 101.2 |
| NRDF-MV | 34.7 | 72.2 |
| Ours | 30.7 | 56.4 |
Revision: Appendix A.5 (Table 11).
[oHvg] asked for the accuracy difference between the identity and full Jacobian at matched steps and hypotheses.
Thank you for this question. To isolate the effect of the approximation, we compared identity and full-Jacobian guidance at the same numbers of steps and hypotheses, and compared their update directions at every flow time. The identity approximation is more accurate than the full Jacobian at the deployed 10 steps and stays within 0.7 mm of it with more steps, while each step is 19 to 24× faster (Table R22). Its update is also a descent direction for the full-Jacobian objective at every flow time (positive cosine in 99.9% of states; Table R23). It therefore keeps the accuracy of the full Jacobian at a small fraction of its cost.
Table R22. Identity vs. full endpoint Jacobian at the same number of steps S. MPJPE (mm) on Mouse-Occluded and Human3.6M-Occluded; last row: guidance time per step. Deployed setting: S=10, N=4.
| Mouse | Human | |||
|---|---|---|---|---|
| Steps S | Identity | Full Jacobian | Identity | Full Jacobian |
| 5 | 11.0 | 13.8 | 34.4 | 53.0 |
| 10 | 10.2 | 10.5 | 30.7 | 38.1 |
| 25 | 10.1 | 9.7 | 29.5 | 31.3 |
| 50 | 10.1 | 9.4 | 29.5 | 29.7 |
| 100 | 9.9 | 9.6 | 30.0 | 29.3 |
| ms / step | 8.6 | 207.4 | 8.7 | 162.9 |
Table R23. Gradient alignment of identity and full-Jacobian guidance (Mouse-Occluded).
| t | Mean cosine | Cosine >0 |
|---|---|---|
| [0.0, 0.2) | 0.47 | 99.8% |
| [0.2, 0.4) | 0.43 | 100.0% |
| [0.4, 0.6) | 0.40 | 100.0% |
| [0.6, 0.8) | 0.46 | 99.9% |
| [0.8, 1.0) | 0.69 | 100.0% |
| All | 0.49 | 99.9% |
Revision: Section 4.3 (Table 5) and Appendix D (Table 27).
[oHvg] asked why the margins over strong baselines are smaller on Human3.6M-Occluded than on mouse.
Thank you for this question. To find where the difference comes from, we compared the quality of the 2D inputs across species, degraded the human input with a less accurate detector, and varied the prior's training data. Relative to body size, mouse detections are about four times less accurate than human ones (Table R24), and Mouse-Occluded occludes every view, whereas Human3.6M-Occluded [2] leaves one of four views unoccluded. Degrading the human input confirms this: with the less accurate HRNet detector, GenTri's gain over RANSAC grows from 21% to 44% (Table R21). Prior quality, by contrast, matters little, since shrinking the prior's training data changes the error only slightly (Table R16). The margins therefore differ mainly because the 2D inputs differ in quality: the worse the detections, the more robust reconstruction can recover, and the smaller human margins reflect cleaner human inputs.
Table R24. 2D input quality relative to body size. Mean back-projected 2D error divided by mean bone length.
| Benchmark | Mean 2D error / bone length |
|---|---|
| Mouse-Occluded | 0.68 |
| Human3.6M-Occluded | 0.16 |
Revision: Appendix A.7 (Table 15).
References
[1] Timothy W. Dunn, Jesse D. Marshall, Kyle S. Severson, et al. Geometric deep learning enables 3D kinematic profiling across species and environments. Nature Methods, 18(5):564–573, 2021.
[2] Laura Bragagnolo, Matteo Terreran, Davide Allegro, and Stefano Ghidoni. Multi-view pose fusion for occlusion-aware 3D human pose estimation. In Computer Vision, ECCV 2024 Workshops, LNCS, pages 117–133. Springer, 2025.
Response to Reviewer oHvg (Part 1/3)
Thank you for your constructive feedback and for recognizing the technical motivation and controlled comparisons. Your comments helped us broaden the evaluation and clarify the method's behavior, as detailed below.
[
oHvg] asked whether the gains hold under naturally occurring occlusion.Thank you for this question. To test GenTri under real occlusion, we recorded four additional open-field sessions with a physical occluder placed in front of one of the four cameras and annotated 100 frames (eight keypoints visible in the unobstructed views). Reconstruction uses the occluded view and two unobstructed views, with no retraining or retuning. As shown in Table R14, GenTri more than halves the error of triangulation, RANSAC and Anipose, is more accurate than multi-view EKS, and improves every paired detector baseline, so its robustness carries over from synthetic to real occlusion. We additionally compare GenTri, RANSAC and multi-view EKS under naturally occurring occlusions in a real cage environment, including a failure case (Figure 4(d) and Figure 7; videos on the anonymous project page).
Table R14. Real occlusion in open-field (OoD). * Mean excludes non-finite predictions or those with errors ≥1 m.
Revision: Table 2 adds the real-occlusion column, Section 4.1 and Appendix B.1 describe the protocol, and Appendix A.1 adds the cage comparisons.
[
oHvg] asked how far lowest-energy selection is from oracle selection, especially under severe occlusion.Thank you for this question. To measure how reliable lowest-energy selection is, we compared it with an oracle that picks the lowest-error hypothesis among the same four candidates, on all frames and on the hardest ones (Table R15). The selected hypothesis stays within 0.2 mm of the oracle on mouse and 1.1 mm on human, and the gap grows only slightly on the hardest frames, far less than the differences between methods. The reconstruction energy thus identifies a near-best hypothesis without ground truth, even under severe occlusion.
Table R15. Lowest-energy vs. oracle selection. MPJPE (mm), N=4.
Revision: Figure 9 and Appendix B.2 (Table 18).