- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies 3D question answering through active viewpoint exploration. Starting from a single rendered image, a VLM moves a virtual camera to gather additional observations before answering. Training consists of supervised fine-tuning on teacher-generated trajectories followed by GRPO, with rewards for answer correctness, tool use, object visibility, and output format. Across six benchmarks, the final model improves average accuracy from 48.6% after SFT to 54.1% after RL. The paper also studies several design choices, including relative camera actions, rendered versus captured images, and supervised initialization.
Critical strengths
The paper addresses a meaningful problem: answering spatial questions often requires obtaining information that is missing from the initial view. The interaction setup makes it possible to study observation selection alongside question answering.
The RL model improves over its SFT initialization on all six benchmarks. The comparisons with general-purpose VLMs under the same interaction interface are also useful and show that the trained model performs competitively in this setting.
Some of the design studies are informative. In particular, the comparison between rendered views and nearby photographs shows the importance of returning observations that match the requested camera pose. The relative-versus-absolute action comparison provides another useful implementation finding.
Critical weaknesses
The novelty over existing active-view methods needs to be better established. CoV already alternates reasoning, camera actions, and rendered observations for 3D QA. Think3D-RL already uses GRPO with terminal answer and format rewards to train multi-step viewpoint selection. The single-view starting condition and access to an external scanned scene distinguish this paper, but I am not yet convinced that these differences establish a substantial methodological advance. Both works are discussed, yet neither is included in the experiments. Comparing against compatible versions of these methods under matched conditions would help establish what the proposed approach adds.
The claim about learning exploration from answer correctness is stronger than the evidence supports. The reward includes a bonus for each tool call in a successful episode and an object-visibility term based on ground-truth 3D annotations. Under the stated weights, twelve calls contribute 0.6, while the two answer-correctness terms contribute at most 0.5. Thus, the objective explicitly encourages longer successful trajectories, even if some calls are redundant. Table 2 examines the tool bonus, but the paper does not report an ablation of the grounding reward or remove both terms together. Learning without intermediate action labels is a reasonable claim; learning useful exploration solely from final-answer correctness has not been demonstrated.
It remains unclear how much of the gain comes from better exploration. RL changes both the observation policy and the model’s ability to answer from those observations. Figure 5 does not separate these effects: grouping questions by the RL policy’s call count does not ensure that SFT and RL receive the same amount of evidence. The qualitative examples are plausible, but do not show whether RL makes better viewpoint choices than SFT or a simple exploration policy. Evaluating SFT and RL answer models on the same collected views, and comparing different exploration policies with a fixed answer model and matched budgets, would make the main conclusion more convincing.
The OOD evaluation needs clarification. Figure 3 includes 7,534 ARKit questions in the RL data, and Section 4.1 confirms that ARKit is used for training. However, Table 2 labels ARKit as OOD, and its improvement contributes substantially to the reported four-benchmark OOD gain. Held-out scenes or question types could still provide a useful generalization test, but that is different from transfer to an unseen dataset domain. The paper should specify these distinctions and document scene overlap across all training and evaluation sources, particularly for the generated ScanNet and ARKit questions. As written, the cross-domain generalization claim is difficult to assess.
Questions for authors
What defines OOD in Tables 1–2? How does the ARKit evaluation set differ from the ARKit training data, and are scenes disjoint across all training and evaluation sources?
How is visibility determined for the grounding reward, particularly when the target is occluded? What proportion of training examples receives this reward, and are results available without both grounding supervision and the tool-call bonus?
In Figure 5, are SFT and RL evaluated on the same questions grouped by the RL policy’s call count? What are the SFT call counts within these groups?
Are all Table 1 outputs scored with the same evaluation prompt and judge settings? Since the same judge model is used during RL and evaluation, has the improvement been checked with an independent judge or a human-reviewed sample?
13 comments
Rebuttal for Reviewer iPE7 2/5
>>> W2 & Q2
The claim of learning exploration from answer correctness is too strong: the tool bonus (12 calls = 0.6 vs. at most 0.5 for correctness) encourages long trajectories, and the grounding term uses ground-truth 3D annotations. How is visibility determined when the target is occluded, what fraction of examples receive the grounding reward, and are there results without both the grounding term and the tool bonus?
>>> A2
In the current reward, the tool bonus is gated by judged answer correctness: , the LLM judge’s verdict on whether the final answer matches the ground truth, acts as an on/off switch for the tool bonus. Answer correctness therefore governs the tool-bonus term as well, so its influence on the reward is not limited to its own, smaller weight.
The grounding reward applies only to questions whose answer object has an annotated 3D location. This covers about 75.6% of the RL corpus (38,981 of 51,543 questions): all ScanQA and curated ScanNet questions, 64.1% of curated ARKit questions, and no MP3D questions. For the remaining questions, the grounding term is zero for every rollout and cancels out of the GRPO advantage, so their updates are driven by the other reward terms, mainly answer correctness. Grounding is 1 if the target objevt falls inside the image boundaries of any view in full trajectrary and is unoccluded (casting a ray from the camera toward the object, if the first mesh hit is more than 0.30 m before the object, it is still considered occluded).
To further quantify the contribution of the grounding reward and the tool-call bonus in Eq. 7, we compare the full-reward policy with two ablations in the mixed-data setting. The first removes the grounding condition while retaining the interaction bonus. The second removes both. We report answer accuracy, the average number of views, and how often the policy brings the annotated target into view. We evaluate all policies on the full benchmark pools (11,522 questions) using the native harness, a budget of 12 interactions, and a sampling temperature of 0.7. As shown in Table R2, removing the grounding reward reduces averaged accuracy from 54.1% to 53.2% and ScanQA target exposure from 95.1% to 91.6%, with both policies averaging 11.0 views per question across the six benchmarks. With both rewards removed, RL still improves accuracy from 48.6% to 53.3% and target exposure from 78.7% to 84.7% over the SFT initialization, with similar average view counts (5.0 vs. 5.3).
Table R2. Grounding-reward ablation in the mixed-data setting. #Views includes the initial view and is averaged across the six benchmarks. ScanQA Target Exposed is the percentage of ScanQA questions for which at least 5% of the target object is visible and unoccluded in any observed view.
| ScanQA | SQA3D | SPB MV | SPB SI | MP3D | ARKit | Avg | #Views | ScanQA Target Exposed (%) | |
|---|---|---|---|---|---|---|---|---|---|
| SFT init | 39.3 | 47.4 | 39.2 | 37.7 | 61.8 | 65.9 | 48.6 | 5.0 | 78.7 |
| Act3D, full reward (ours) | 42.5 | 47.7 | 50.5 | 41.2 | 62.9 | 79.6 | 54.1 | 11.0 | 95.1 |
| Act3D, no grounding reward | 41.8 | 47.8 | 45.4 | 40.3 | 64.7 | 79.4 | 53.2 | 11.0 | 91.6 |
| Act3D, no grounding reward, no tool bonus | 38.9 | 47.2 | 48.1 | 39.1 | 64.7 | 81.6 | 53.3 | 5.3 | 84.7 |
| Act3D, maximize tool bonus when grounded | 41.2 | 48.3 | 48.8 | 41.3 | 67.1 | 80.4 | 54.5 | 8.4 | 91.0 |
Tool bonus supports our goal of active exploration that yields higher capture of targeted object (+) more than pure accuracy gain. If we also want to further optimize efficiency, we experimented with one more reward interaction: tool bonus credited only with a correct answer, but verified grounding now acts as a shortcut to the full credit rather than as a separate bonus:
\begin{cases} N_{\max}, & R_{\mathrm{gnd}}=1, \cr \min\lbrace n_{\mathrm{int}},N_{\max}\rbrace , & \text{otherwise,} \end{cases}$$ where $n_{\mathrm{int}}$ is the number of interactions in $\tau$, and $R_{\mathrm{gnd}}\in\lbrace 0,1\rbrace$ uses the annotated 3D location of the object of interest, taking the value 1 when that object is visible in some rendered view of $\tau$. A rollout that is judged correct *and* has actually brought the target object into view therefore receives the full credit $N_{\max}$ at once, after as few as a single call. The result for "Act3D, maximize tool bonus when grounded" in Table R2 shows that gives matching or slightly better average accuracy over the current policy (**54.5% vs. 54.1%**), while using fewer views (**8.4 vs. 11.0**) at a lower exposure (91.0% vs. 95.1%). This suggests that tying the tool bonus to grounding encourages more efficient exploration. Yet, higher accuracy at lower target exposed rate may be attributed to better answering in low-exploring problem as it contains shorter interleaved chat history. We will include this result in the revised paper.Rebuttal for Reviewer iPE7 3/5
>>> W3 & Q3
It's unclear how much of the gain comes from better exploration. Fig. 5 groups by the RL policy's call count, so SFT and RL don't receive the same evidence. Evaluate SFT and RL answerers on the same views, and compare exploration policies with a fixed answerer and matched budget. In Fig. 5, are SFT and RL evaluated on the same questions, and what are SFT's call counts within each group?
>>> A3
Yes. In Fig. 5, the base, SFT and RL models are evaluated on the same 4,649 ScanQA validation questions, grouped by the RL policy's number of moves. To avoid a grouping that depends on any particular policy's behaviour, we instead compute, from scene geometry alone, the number of interactions needed to bring the answer object into view from the initial pose: ⌈|bearing| / 30°⌉ default turns (at least 1), plus forward moves if the object is still more than 2.5 m away. Table R11 below re-plots Fig. 5 with this new grouping.
Under this scene-grounded grouping, SFT makes 2.0–3.5 calls in groups 2–6 and 5.4 in group 7+, rising with difficulty but well below the number of interactions needed in groups 3 and above. RL’s gain over SFT is therefore largest where SFT under-explores most: +4.4 to +5.4 points in groups 3, 4 and 6, and +10.9 in group 7+, where the object is farthest from the initial view.
Table R11. Change in accuracy relative to the base model on ScanQA validation, grouped by the number of interactions needed.
| Interactions needed | SFT − base | RL − base | RL − SFT |
|---|---|---|---|
| 2- | −1.1 | +0.8 | +1.9 |
| 3 | +4.0 | +8.4 | +4.4 |
| 4 | +5.6 | +11.0 | +5.4 |
| 5 | +5.5 | +8.0 | +2.5 |
| 6 | +6.5 | +11.3 | +4.8 |
| 7+ | +5.4 | +16.3 | +10.9 |
To further isolate the effect of interaction in a controlled way, we conducted two analyses.
First, we measure whether the policy finds relevant objects using ScanQA target exposure, the percentage of questions for which at least 5% of the annotated target object is visible and unoccluded in any observed view during the model's inference rollout.
Table R3. Answer accuracy and target exposure on the full benchmark pools. #Views counts the initial view and all views returned by interactions, averaged across the six benchmarks. The initial view alone exposes the target in 50.6% of ScanQA questions.
| Model (active) | ScanQA | SQA3D | SPB MV | SPB SI | MP3D | ARKit | Avg | #Views | ScanQA Target Exposed (%) |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-9B (base) | 36.7 | 33.5 | 21.3 | 24.7 | 30.9 | 40.1 | 31.2 | 5.4 | 62.9 |
| Qwen3.5-27B | 27.5 | 40.2 | 33.2 | 31.4 | 33.8 | 33.8 | 33.3 | 7.0 | 76.6 |
| InternVL3.5-38B | 30.9 | 41.5 | 43.2 | 41.6 | 50.1 | 55.8 | 43.9 | 2.1 | 55.1 |
| GLM-4.6V | 30.6 | 42.1 | 43.6 | 39.5 | 40.4 | 45.8 | 40.3 | 5.3 | 67.5 |
| Gemini 3.6 Flash | 39.1 | 53.8 | 45.5 | 44.4 | 53.4 | 73.0 | 51.5 | 7.7 | 89.5 |
| Gemini 3.1 Pro | 40.7 | 52.7 | 50.3 | 43.9 | 64.4 | 72.2 | 54.0 | 8.5 | 89.7 |
| SFT init (mixed) | 39.3 | 47.4 | 39.2 | 37.7 | 61.8 | 65.9 | 48.6 | 5.0 | 78.7 |
| Act3D (mixed, ours) | 42.5 | 47.7 | 50.5 | 41.2 | 62.9 | 79.6 | 54.1 | 11.0 | 95.1 |
Second, we hold the answerer fixed and vary only the explorer that collects the views. On a 300-question-per-benchmark subset, we give Gemini 3.6 Flash the question and the view sequence collected by each explorer. Each explorer retains its original trajectory length. As shown in Table R6, the fixed answerer achieves its highest mean accuracy with Act3D’s views (52.2%), compared with 49.5% using views from the SFT initialization and 49.8% using views collected by Gemini 3.6 Flash itself. Act3D’s views improve accuracy over SFT views on all six benchmarks. With the answerer held fixed, these gains show that the observations collected by the RL policy provide more useful evidence for answering.
Table R6. QA accuracy (%) with Gemini 3.6 Flash as the fixed answerer on a 300-question-per-benchmark subset.
| Explorer (views from) | ScanQA | SQA3D | SPB MV | SPB SI | MP3D | ARKit | Avg |
|---|---|---|---|---|---|---|---|
| Act3D (RL) | 36.3 | 55.0 | 43.4 | 34.4 | 63.0 | 81.0 | 52.2 |
| SFT init | 35.0 | 53.0 | 35.7 | 31.5 | 62.3 | 79.7 | 49.5 |
| Base Qwen3.5-9B | 31.3 | 47.7 | 32.9 | 26.6 | 59.0 | 73.0 | 45.1 |
| Gemini 3.6 Flash | 35.3 | 51.7 | 34.6 | 36.4 | 62.7 | 78.3 | 49.8 |
| Random walk | 31.0 | 46.0 | 29.2 | 31.7 | 60.3 | 74.0 | 45.4 |
| Fixed turn-in-place sweep | 35.0 | 51.7 | 32.9 | 33.9 | 60.3 | 79.3 | 48.9 |
Rebuttal for Reviewer iPE7 1/5
We thank the reviewer for the in-depth review. We appreciate the recognition that the paper addresses a meaningful problem, that the RL policy improves over its SFT initialization on all six benchmarks and is competitive with general-purpose VLMs under the same interaction interface, and that the design studies on rendered versus captured views and relative versus absolute actions are informative. We address each point below.
>>> W1>>> A1Thanks for raising this concern. Below, we compare Act3D with CoV and Think3D-RL in detail to better establish our contribution and novelty. The key distinction between these approaches and ours lies in their assumption that complete 3D information is provided as input to the model. Under this assumption, both Think3D-RL and CoV start from a set of input images. Specifically, Think3D-RL takes multiple images as input and uses Pi3 to reconstruct a 3D point cloud. Renderings of this point cloud can reveal details from new perspectives, but the available evidence is bounded by what Pi3 recovers from the input images. Think3D-RL selects from a finite set of overhead viewpoints at the corners of the scene (in 72.0% of the tasks we tested below, it chooses the same view: 0° azimuth, looking down at 60°), whereas Act3D supports a continuous range of camera poses (free exploration). CoV similarly has access to prerecorded video frames and even a bird’s-eye view of the scene before refining the camera position and orientation. These initial inputs expose 99.7% of the target objects in the ScanQA validation set. Furthermore, CoV is a training-free, prompt-based framework and therefore relies on the underlying model’s existing ability to explore and reason in 3D. As a result, CoV cannot equip the model with an intrinsic capability to explore and search in 3D space.Act3D formulates 3D reasoning under partial observability and exposes the scene solely as an interactive environment. Each rollout begins with a single egocentric image. Receiving only the resulting first-person renderings, the model must iteratively navigate and reorient the camera to acquire new information, while reasoning about the spatial relations between observations to answer the question. Through interaction with this otherwise hidden 3D environment, Act3D learns when and how to acquire missing visual evidence.
To further address your concern, we compare against these methods under strictly matched conditions. Specifically, we evaluated Act3D, Think3D-RL, and CoV on the same six 3D QA benchmarks and measured target exposure on ScanQA to assess how often each method brought the target object into view. Act3D achieved the highest accuracy on all six benchmarks, averaging 54.1%, compared with 41.4% for Think3D-RL and 41.9% for CoV. On ScanQA, Act3D exposed 95.1% of the target objects, compared with 86.8% for Think3D-RL and 33.3% for CoV. The exposure results show that Act3D’s learned policy more frequently acquires task-relevant visual evidence.
Comparison with Think3D-RL and CoV under matched conditions.
* Instead of running Think3D-RL's official pipeline, which starts from 3D reconstruction with Pi3, we skip reconstruction and give it the ground-truth 3D mesh, so that its results are comparable to ours in terms of exploration ability. Its ScanQA target exposure is high despite requesting only 0–2 viewpoints, because its selectable viewpoints include overhead views.
** CoV's official evaluation uses privileged starting input, including a bird's-eye view, up to 15 viewpoints of the scene, and a starting-view selector. For comparability, we reduced the starting view to a single viewpoint and removed the viewpoint selector.