- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses the problem of interpretation of information in visual cortex, by trying the decoding of spatial 3D layout of rooms from fMRI scans; authors propose a framework that take input the fMRI signals together with camera trajectories, and extract an approximation of 3D layout; it is sort of SLAM through fMRI decoding. They showcase promising initial results, and compare with respect to baselines employing modern fMRI decoding technologies.
Critical strengths
-
The paper contains an interesting formulation, and addresses a challenging problem, worth of exploration: fMRI decoding for spatial understanding. The implications for future research are evident, and it is particularly relevant for the ICLR community
-
The manuscript is well written, well organized, and pleasant to read
-
The proposed solution is technically sound and well implemented.
-
The reported results appear promising even if far from practical validity.
Critical weaknesses
My main concerns are the following, and in this status, I am not yet convinced this paper reaches the bar for ICLR:
-
Not convincing final output: authors claim in the title that the framework is able to recover 3D layouts, but the output is a noisy uncomplete point cloud. I would have expected a structured output, also in the form of simple CAD polygonal representations, that could have been obtained through post-processing, and it could have been compared to methods already trying to solve this task, like AtlantaNet$^1$ or Deep3DLayout$^2$. Even the comparison, could have considered specific metrics for 3D layouts, like 3D IoU.
-
Practical limitations: I cannot imagine a practical scenario where camera trajectories are available together with fMRI, apart of the synthetic experiments considered. At this point, I would have considered a simpler scenario of omnidirectional exploration of indoor panoramic scenes, in which only the orientation is considered, and this would release the need of camera trajectories, and would provide a more controlled setting
[1] Pintore, Giovanni, Marco Agus, and Enrico Gobbetti. "AtlantaNet: inferring the 3D indoor layout from a single 360∘ image beyond the Manhattan world assumption." European conference on computer vision. Cham: Springer International Publishing, 2020.
[2] Pintore, G., Almansa, E., Agus, M., & Gobbetti, E. (2021). Deep3dlayout: 3d reconstruction of an indoor layout from a spherical panoramic image. ACM Transactions on Graphics (TOG), 40(6), 1-12.
Questions for authors
A post-processing component able to provide a structured 3D layout, and the corresponding benchmark would affect my assessment.
4 comments
Re: Layout decoding
Thank you for the close look at the plans and for both suggestions.
Manhattan constraint and redundant edges. Yes, both come from our post-processing, which treats every source and the ground truth alike. The rule traces each plan on a 0.25 m grid aligned with two axes, so every wall is parallel to an axis. A wall whose predicted distance changes between views turns into steps, and the rule merges only edges shorter than 1 m. True depth gets the same treatment: with the constant axes of the figures, its plans have a median of 8 corners per route and 14 per video, against 6 and 16 for DRAFT. All ten apartments are axis aligned, so the constraint fits here, but slanted walls, which AtlantaNet handles, would break it. Without the rule, DRAFT stays below the template: raw footprints scored against raw true footprints give route 3D IoU 0.386 against 0.415.
Reprojected layout. Following your suggestion, the new Figure 13 (page 57) draws the plans into the views, as layout papers do on the panorama. It marks the visible floor, ceiling and corner edges of the true plan (white) and the DRAFT plan (pink) in views 1, 4 and 8 of a route, beside top views before and after the rule. The figure also shows a limit of the rule. A route's plan outlines all the free space seen along it, so inner walls vanish, and in the first row view 1 ends up outside the merged true plan.
Different levels of complexity. The new Table 31 (page 56) groups the 324 routes by the corners of their true plan, as layout benchmarks do. For each group, Figure 13 shows the route with the lower-median DRAFT IoU, a rule we fixed before drawing.
| Route 3D IoU | 4 corners | 6 | 8 | 10 or more |
|---|---|---|---|---|
| Routes | 57 | 87 | 77 | 103 |
| True depth, same rule | 0.917 | 0.916 | 0.916 | 0.915 |
| DRAFT | 0.397 | 0.399 | 0.398 | 0.357 |
| Fixed template | 0.375 | 0.416 | 0.432 | 0.395 |
DRAFT drops at 10 or more corners while true depth stays at 0.92, and it beats the template only on four-corner routes (+0.022, higher on 34 of 57). This grouping is post hoc and uses the planned axes from camera headings. Constant axes give the same pattern (Table 31).
Does Figure 13 show what you had in mind? If another formulation, such as plans without the axis constraint, would weigh more in your assessment, tell us and we will add it before the discussion closes.
Thanks for additional results
According to the results provided, the proposed solution appears to be far from practical utility, and the problem is far from being solved. Nevertheless the problem itself and the methodology is intruiguing to my opinion, so I feel it can be a bordelrline accept.
Re: Thanks for additional results
Thank you for staying with the discussion and for updating your review. We agree that the method is far from practical use and that the problem is far from solved. The revised paper states both, in the abstract and the limitations. The final version will keep the floor-plan benchmark, Figure 13 and Table 31, which came from your suggestions.
Layout decoding
Thanks for providing additional results in Section 5.5. I noticed a curious pattern related to the generated layout (they all seem to be with Manhattan constraint, and with redundant number of edges). Is that an effect post-processing? Authors should also probably show the reprojected layout according to original formulations, and consider examples with different level complexity, if available.