acceptodds
Under review as a conference paper at ICLR 2027

DRAFT: Decoding and Reconstructing 3D Room Layouts from fMRI and Camera Trajectories

Abstract

While functional magnetic resonance imaging (fMRI) decoders reconstruct images and 3D objects, recovering the metric geometry of rooms from a moving viewpoint remains an open challenge. We introduce, to our knowledge for the first time, the task of reconstructing 3D layouts of held-out scenes using only the known camera trajectories and the fMRI signals of participants watching indoor walkthroughs, without any visual frames. The task is hard because of the temporal mismatch between rapid visual changes and blurred fMRI signals. Our method, DRAFT (Decoding and Reconstructing 3D room lAyouts from fMRI and Trajectories), extracts a compact layout representation from the early visual cortex and fuses this neural code with the geometric free space derived from camera motion. To evaluate it, we propose layout r, the Pearson correlation between predicted and ray-cast log depths. On ten held-out scenes (20 participants), DRAFT reaches a layout r of 0.480, outperforming the camera path alone (0.446) and seven fMRI decoders (at most 0.429). DRAFT also works as a plug-in that improves all seven decoders without retraining, raising their layout r from 0.171–0.429 to 0.487–0.493. Brain activity and camera motion can thus recover the metric layout of unseen rooms, a first step toward reconstructing the 3D spaces people see.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI decoding, brain decoding, visual cortex, naturalistic video, room layout, 3D reconstruction, depth estimation, camera trajectory, flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V1x5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the problem of interpretation of information in visual cortex, by trying the decoding of spatial 3D layout of rooms from fMRI scans; authors propose a framework that take input the fMRI signals together with camera trajectories, and extract an approximation of 3D layout; it is sort of SLAM through fMRI decoding. They showcase promising initial results, and compare with respect to baselines employing modern fMRI decoding technologies.

Critical strengths
  1. The paper contains an interesting formulation, and addresses a challenging problem, worth of exploration: fMRI decoding for spatial understanding. The implications for future research are evident, and it is particularly relevant for the ICLR community

  2. The manuscript is well written, well organized, and pleasant to read

  3. The proposed solution is technically sound and well implemented.

  4. The reported results appear promising even if far from practical validity.

Critical weaknesses

My main concerns are the following, and in this status, I am not yet convinced this paper reaches the bar for ICLR:

  1. Not convincing final output: authors claim in the title that the framework is able to recover 3D layouts, but the output is a noisy uncomplete point cloud. I would have expected a structured output, also in the form of simple CAD polygonal representations, that could have been obtained through post-processing, and it could have been compared to methods already trying to solve this task, like AtlantaNet$^1$ or Deep3DLayout$^2$. Even the comparison, could have considered specific metrics for 3D layouts, like 3D IoU.

  2. Practical limitations: I cannot imagine a practical scenario where camera trajectories are available together with fMRI, apart of the synthetic experiments considered. At this point, I would have considered a simpler scenario of omnidirectional exploration of indoor panoramic scenes, in which only the orientation is considered, and this would release the need of camera trajectories, and would provide a more controlled setting

[1] Pintore, Giovanni, Marco Agus, and Enrico Gobbetti. "AtlantaNet: inferring the 3D indoor layout from a single 360∘ image beyond the Manhattan world assumption." European conference on computer vision. Cham: Springer International Publishing, 2020.

[2] Pintore, G., Almansa, E., Agus, M., & Gobbetti, E. (2021). Deep3dlayout: 3d reconstruction of an indoor layout from a spherical panoramic image. ACM Transactions on Graphics (TOG), 40(6), 1-12.

Questions for authors

A post-processing component able to provide a structured 3D layout, and the corresponding benchmark would affect my assessment.

4 comments
Reviewer_V1x5OpenReview ↗

Layout decoding

Thanks for providing additional results in Section 5.5. I noticed a curious pattern related to the generated layout (they all seem to be with Manhattan constraint, and with redundant number of edges). Is that an effect post-processing? Authors should also probably show the reprojected layout according to original formulations, and consider examples with different level complexity, if available.

AuthorsOpenReview ↗

Re: Layout decoding

Thank you for the close look at the plans and for both suggestions.

Manhattan constraint and redundant edges. Yes, both come from our post-processing, which treats every source and the ground truth alike. The rule traces each plan on a 0.25 m grid aligned with two axes, so every wall is parallel to an axis. A wall whose predicted distance changes between views turns into steps, and the rule merges only edges shorter than 1 m. True depth gets the same treatment: with the constant axes of the figures, its plans have a median of 8 corners per route and 14 per video, against 6 and 16 for DRAFT. All ten apartments are axis aligned, so the constraint fits here, but slanted walls, which AtlantaNet handles, would break it. Without the rule, DRAFT stays below the template: raw footprints scored against raw true footprints give route 3D IoU 0.386 against 0.415.

Reprojected layout. Following your suggestion, the new Figure 13 (page 57) draws the plans into the views, as layout papers do on the panorama. It marks the visible floor, ceiling and corner edges of the true plan (white) and the DRAFT plan (pink) in views 1, 4 and 8 of a route, beside top views before and after the rule. The figure also shows a limit of the rule. A route's plan outlines all the free space seen along it, so inner walls vanish, and in the first row view 1 ends up outside the merged true plan.

Different levels of complexity. The new Table 31 (page 56) groups the 324 routes by the corners of their true plan, as layout benchmarks do. For each group, Figure 13 shows the route with the lower-median DRAFT IoU, a rule we fixed before drawing.

Route 3D IoU4 corners6810 or more
Routes578777103
True depth, same rule0.9170.9160.9160.915
DRAFT0.3970.3990.3980.357
Fixed template0.3750.4160.4320.395

DRAFT drops at 10 or more corners while true depth stays at 0.92, and it beats the template only on four-corner routes (+0.022, higher on 34 of 57). This grouping is post hoc and uses the planned axes from camera headings. Constant axes give the same pattern (Table 31).

Does Figure 13 show what you had in mind? If another formulation, such as plans without the axis constraint, would weigh more in your assessment, tell us and we will add it before the discussion closes.

Reviewer_V1x5OpenReview ↗

Thanks for additional results

According to the results provided, the proposed solution appears to be far from practical utility, and the problem is far from being solved. Nevertheless the problem itself and the methodology is intruiguing to my opinion, so I feel it can be a bordelrline accept.

AuthorsOpenReview ↗

Re: Thanks for additional results

Thank you for staying with the discussion and for updating your review. We agree that the method is far from practical use and that the problem is far from solved. The revised paper states both, in the abstract and the limitations. The final version will keep the floor-plan benchmark, Figure 13 and Table 31, which came from your suggestions.

Reviewer_ZGd5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method for 3D room layout reconstruction from the fMRI signals acquired from a person watching first-person videos of the room and the associated camera trajectory. "Layout" here appears to mean the 3D point cloud of the unfurnished room. The input fMRI signals are (I believe) associated with a set of video recordings, where the same room has different material, color, and furniture, but the same layout and same camera trajectory. The approach is to have ridge regressors map the early visual cortex (EVC) signals and camera trajectory information to a pre-defined "layout code" from which a depth map is rendered, from which a flow matching model generates a detailed depth map. This is combined with empty-space information from the camera trajectory to predict the final depth map at each point on the trajectory. The listed contributions are (a) a learned function that maps the EVC and camera motion to layout codes, and (b) the DRAFT layout that fuses both sets of information together. The approach is evaluated on 10 indoor scenes with multiple participants, and demonstrates that the reconstructed rooms are more accurate than an input-free generated template room.

Critical strengths
  1. Interesting and evocative. The task is, frankly, rather interesting, and not something I ever thought I'd read about. This alone made it an engaging read.

  2. The argument that the camera trajectory contains a significant amount of information about the geometry of a scene is a sound one, and is indeed something that has been explored in the computer vision community.

  3. Approach. The overall idea of the approach, to solve the coarse-level problem first, then fill in the details using a generative model, is sound and makes sense.

Critical weaknesses
  1. Motivation. No motivation is given for why this problem is important to solve. The reader has no way to assess the relevance of the task and the proposed approach. I expect there is a good reason for doing this, but it is not to be found in the paper.

  2. Inputs. The fMRI signals from a person watching a set of videos and the associated camera trajectory is a rather unusual pair of inputs. In what real application would this ever occur; why would we want to reconstruct rooms this way? If not an application, what scientific information does this afford us? Is the intention that someone would walk around with an IMU and some device to measure brain signals and the proposed approach would produce a 3D model of the scene? If so, why not just use a camera?

    1. The inputs are also somewhat under-specified: are the videos from head-mounted cameras, or handheld, or arbitrary camera trajectories? If head-mounted, this should encode much more information about the scene via the head pose, which has been shown to give extra cues beyond just free space.
    2. The pooled recordings are also a bit unclear: what exactly is changing, and what's the rationale for this?
  3. Approach. Much of the approach is underspecified (see below), but from what I understand, the authors propose to train a flow-matching model (Fig 2b) to fill in details from the coarse depth map. However, there are only 10 scenes. Even with leave-one-scene-out cross-validation, this is a tiny amount of data to train a flow-matching model - it will certainly be overfitting to the training data. This would only be useful for the performance of the method if all rooms have a similar layout. Relatedly, the room diversity is not specified, so it is unclear how much variation there is anyway.

  4. Clarity. As stated in the AI Use Statement, "We used large language model assistants extensively, under the authors’ supervision. They helped write and debug the analysis, model and evaluation code, develop the method, and propose, refine, preregister, run and audit experiments and analyses. They also helped interpret results, search the literature and verify references, translate the authors’ notes into English, and draft and edit the text, equations and figures." This is unfortunately to the significant detriment of the paper - it is exceptionally difficult to follow the paper and evaluation. The LLM is doing a very poor job of communication, I am sure the authors could do better themselves. The evaluation is particularly hard to follow; see, for example, Table 3 which is very hard to parse. There are also several very odd remarks throughout the paper, which read like experimental notes that should never have appeared in the paper:

    1. "An earlier implementation compared the two sides of the penalty in different coordinates, so all main results use ten generators retrained with the corrected loss" [L248]
    2. "We added this step and its selection after earlier results on these scenes" [L311]
    3. "The calibration factor was fixed after earlier results on these scenes" [L432]
  5. Evaluation.

    1. It appears from Table 2 that the proposed approach "DRAFT (own generator)" performs worse than CLIP retrieval, Brain-Diffuser, and MindEye combined with the proposed DRAFT layout. From this, the reader concludes that the proposed approach is sub-optimal, but a component of it is helpful when combined with other methods.
    2. From Table 1 and Fig 3, it appears that the fMRI information adds very little to the performance: most of the signal comes from the camera trajectory. Since this is a somewhat odd pairing of inputs, it is important to evaluate the camera-free performance as well. L374 takes a step towards this, but a retraining without this signal is what is actually required to show benefit.
    3. It is unclear whether the results are evaluated with respect to the furnished or unfurnished ground truth.
  6. Missing methodological details. The method specification is highly incomplete and would not be reproducible from the paper alone. My copy has many symbols circled with the annotation "undefined", for example. Some examples include:

    1. L226: this section on the renderer says what it does, in words, but not how. Is it a neural renderer, a graphics renderer, a function?
    2. L210: "12 camera motion regressors" are undefined.
    3. L256: "a ridge regression on these and and other path features", the latter are undefined.
    4. L243: "The network is a 3D U-Net (Çiçek et al., 2016) over space and views" [L243], what is meant by "over space and views"?
Questions for authors
  1. What is the significance of this task?

  2. In what scenario would the specified inputs be available (and not, for example, a video)?

  3. How is overfitting avoided for the flow-matching model?

Reviewer_jYpvOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the reconstruction of metric 3D room layouts and surfaces of unseen indoor walkthrough scenes using only fMRI brain activity and known camera trajectories, without access to visual images. The authors propose DRAFT, a coarse-to-fine reconstruction framework. First, a 22-dimensional layout code is decoded from early visual cortex (EVC) activity and camera motion features using ridge regression. This code is rendered into coarse depth maps, which are subsequently refined into eight consecutive depth frames using a flow-matching-based 3D U-Net. Meanwhile, free-space features extracted from camera trajectories are used to predict a 3×33\times3 trajectory-based layout, which is fused with the brain-decoded representation to produce the final 3×33\times3 DRAFT layout. This layout replaces the low-spatial-frequency components of the depth maps, which are then converted into 3D point-cloud surfaces through range-scaling calibration (a=2a=2). Experiments on 10 held-out scenes show that DRAFT achieves a layout correlation of r=0.480r=0.480, outperforming the trajectory-only baseline (r=0.446r=0.446) and seven adapted fMRI decoders (best r=0.429r=0.429). DRAFT can also serve as a plug-in module to improve the layout correlation of all baseline decoders. Furthermore, the main paper and its extensive 28-page appendix provide detailed discussions of the limitations of 3D surface metrics, hyperparameter selection bias, generalization to five additional held-out scenes, and mediation analyses involving low-level visual features.

Critical strengths
  1. Novel and meaningful research problem. The paper extends fMRI-based visual decoding from images, videos, and individual objects to 3D indoor scene reconstruction under continuous viewpoints. This challenging task provides a valuable research direction at the intersection of brain decoding and 3D vision.
  2. Reasonable and generalizable method design. The proposed method combines a low-dimensional Layout Code with trajectory-derived Free-space Priors to integrate neural information and geometric constraints. By separating coarse layout from depth details, the predicted layout can also serve as a standalone module for existing fMRI decoders without retraining, offering practical flexibility and extensibility.
  3. Comprehensive experimental evaluation. The paper includes comparisons with seven fMRI decoders, component ablations, neural input controls, and evaluations across multiple metrics. Additional experiments examine depth calibration, incorrect scene predictions, camera trajectory perturbations, and generalization to additional scenes. The inclusion of unexpected results further improves the transparency and credibility of the experimental findings.
Critical weaknesses
  1. Limited performance improvement. Despite extensive experimental validation, DRAFT achieves a Layout rr of 0.480, only 0.034 higher than the trajectory-only baseline (0.446). This modest gain suggests that much of the performance comes from geometric priors rather than fMRI signals. Given the complexity of the proposed method, the additional benefit remains limited.

  2. Limited improvement in final 3D reconstruction quality. The gains in coarse layout prediction do not translate into clear improvements in 3D surface reconstruction. DRAFT shows no substantial advantage in F-score or Chamfer Distance. For example, its F@25cm (0.484) is not significantly better than Generator alone (0.478) and remains below the original Brain-Diffuser and MindEye results.

  3. Sensitivity to evaluation metrics and post-processing. The primary metric, Layout rr, evaluates relative depth relationships within a 3×33\times3 grid, which closely aligns with the output of the proposed Fusion Head but does not fully reflect metric 3D reconstruction accuracy. Moreover, surface reconstruction performance is sensitive to Range Calibration. Without calibration, DRAFT performs worse than Generator alone, and even with method-specific calibration, it does not achieve the best reconstruction results. These observations raise concerns about the robustness of the reported improvements.

Questions for authors

Questions for Authors

1. What unique geometric information do brain signals provide beyond camera trajectories?

Although fMRI improves layout prediction, it remains unclear what information it contributes that cannot be inferred from camera trajectories alone. Could the authors clarify the complementary roles of these two inputs?

2. Do brain signals encode 3D geometry beyond low-level visual features?

Appendix J.2 shows no significant additional gain from EVC after controlling for low-level image features. How should this result affect the interpretation of the geometric information decoded from brain activity?

3. Why do improvements in layout prediction not translate into better 3D surface reconstruction?

Despite improvements in layout r, the gains in surface reconstruction metrics remain limited. Could the authors explain this discrepancy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.