acceptodds
Under review as a conference paper at ICLR 2027

EgoPC: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video

Abstract

Answering complex spatial reasoning questions from egocentric video requires determining what evidence is needed, acquiring targeted observations, and combining them through multiple reasoning steps. Existing vision-language models (VLMs) struggle to perform this process reliably. Agentic systems address this challenge by querying external vision and geometry tools, but training their reasoning leaves these tools unchanged, so inaccurate observations can still lead to incorrect answers. We introduce EgoPC, a framework that internalizes a plan-perceive-compose process within a single VLM. To this end, we first train specialized Planner and Perception teachers using separate adapters (Dual-LoRA), then fine-tune the corresponding experts in a mixture-of-experts (MoE) student under their supervision. We subsequently train the student on trajectories generated by its current policy using a segment-level extension of multi-teacher on-policy distillation (MOPD), assigning different trajectory segments to different supervisors. The Planner teacher supervises which evidence-seeking questions to ask, when to stop gathering evidence, and how to compose the final answer. The Perception teacher supervises the video-grounded answers to those questions. At inference, the student executes this adaptive workflow end-to-end, without external spatial tools. Across three spatial reasoning benchmarks, EgoPC students built from Qwen3-VL-8B and SenseNova-SI-1.3-8B outperform their base models by an average of 9.2 and 7.5 percentage points, respectively. Across both backbones, EgoPC reduces inference FLOPs by 65.8–66.3% and achieves 1.29–1.49 speedups over their shared Dual-LoRA teacher.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
egocentric spatial reasoning, video understanding, vision-language models, knowledge distillation, agentic reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bUDHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoP²C proposes a planning-perception–combination framework for spatial intelligence tasks. This method trains a combination teacher and a perception teacher separately, and then, through multi-teacher online policy distillation, internalizes the interaction process between the teachers into a single student model with role-specific experts. Experiments on three benchmarks demonstrate that this method consistently improves the performance of the base model while reducing the inference overhead compared to teacher-based systems.

Critical strengths
  1. The problem is well-defined, and the method is coherently designed. Separating evidence extraction from answer generation aligns with the characteristics of first-person spatial reasoning tasks; the model architecture and training process are also structured around this division of labor.

  2. The performance improvements are practically significant. Compared to a direct-answering baseline trained on the same Q&A data, a performance advantage persists. The student model based on SenseNova achieves improvements of 6.5 and 4.4 percentage points on EgoProx and R3D-SQA, respectively, compared to the direct SFT+GRPO baseline.

Critical weaknesses
  1. The impact of weakly supervised perceptual rewards on the final LLM policy requires further clarification. In actual inference, paths that are helpful for arriving at the correct answer are not necessarily the same as paths where the perceived result is correct. Can the authors quantitatively characterize, on a subset of data, whether weak reward optimization improves perceptual accuracy in the intermediate space? Furthermore, how common are cases where the intermediate observation is incorrect but the final answer is correct?

  2. I noticed that, across the three datasets, the performance improvement on vsi-bench was significantly smaller than on the other two. Should the authors clarify whether this method exhibits a certain preference for specific datasets, or explain why it achieved only a limited improvement on vsi-bench?

  3. The sources and scope of efficiency gains remain unclear. Clearly, the reasoning paths and strategies of student models and teacher models differ; I hope to more clearly identify the specific efficiency gains achieved after distillation. For example, could we decompose the reasoning cost into visual encoding, pre-filling, and decoding, and compare the impact of cache reuse under the same interaction trajectories to clarify the contribution of each design aspect to the efficiency gains?

Questions for authors

See Weakness.

Overall, I have a positive view of the work presented in this paper.

2 comments

Thank you to the author for the rebuttal. I stand by my positive review.

Thank you very much for your follow-up. We sincerely appreciate your time and your positive assessment.

Reviewer_iW9DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoP2C\text{EgoP}^2\text{C}, a framework that internalizes the ``plan-perceive-compose'' multi-step reasoning process into a single MoE-based vision-language model for egocentric video spatial reasoning. To circumvent gradient conflicts between heterogeneous tasks, the authors train decoupled Dual-LoRA teachers and distill them into role-specific MoE layers using segment-level multi-teacher on-policy distillation (MOPD). The effectiveness and inference efficiency of the approach are verified through performance comparisons, ablation studies, and computational profiling on three egocentric spatial benchmarks (VSI-Bench, EgoProx, and R3D-Bench).

Critical strengths

Clear and Well-Motivated Task Formulation: In egocentric video understanding, visual evidence is inherently scattered across temporal frames, and compositional spatial queries naturally demand determining what intermediate measurements to acquire before deciding how to synthesize them. The paper provides a clear, explicit decomposition paradigm (plan-perceive-compose) tailored to these challenges, offering a principled structure for complex multi-step spatial reasoning.

Critical weaknesses

Limited Benchmark and Backbone Diversity: The empirical evaluation is evaluated on a relatively narrow selection of benchmarks and base model architectures. The current experiments are insufficient to demonstrate whether the proposed method remains effective across diverse model families, scaling regimes, and varied reasoning task distributions.

Unsubstantiated and Potentially Misleading Claim on Internalized Perception vs. External Tools: The paper motivates its approach by stating that external vision/geometry tools in agentic pipelines remain static during reasoning training, causing error propagation. However, external tools can be upgraded, calibrated, or replaced independently, often achieving higher geometric and metric precision than an internalized VLM. This argument alone is insufficient to justify why internalizing perception is fundamentally superior, and the paper lacks rigorous comparative experiments validating that internalized perception outperforms state-of-the-art external toolkits. This framing could be potentially misleading regarding the optimal paradigm choice.

Questions for authors

How does the proposed method perform on other comprehensive video benchmarks (e.g., MMSI-Video-Bench)?

How does the framework generalize when applied to alternative base model architectures or different model scales?

Reviewer_FMvZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses adapting LVLMs for VQA. While many other methods rely on external tool calls, these tools may give incorrect answers and it is difficult to correct for this (RL with tools is hard since it remains unclear whether the problem is in the answer of the tool or in the reasoning/tool call formulation). Instead, this paper proposes to specialize an LVLM to answer perceptual queries (replacing visual too calls). It also specializes an LVLM as a planner.

More specifically, this paper starts from an existing LVLM (Qwen3-VL and SenseNova). It then trains a 'Planner' and a 'Perception' variant. In its simplest form, the 'Planner' first determines whether to call the original LVLM directly (no intermediate steps) or if it needs to go into planning mode. In planning mode, it calls the 'perception' variant to gather visual evidence before answering the question. They train both models with LoRA adaptation using Reinforcement Learning and SFT. Then these models acts as teachers but these can also be directly used to solve the problem and is their Dual-LoRA approach (Table 1). They then distill these models into a single mixture-of-experts student using again reinforcement learning. This is the main EgoP2C model. Finally, they demonstrate additional SFT finetuning of this model improves results even further (Tab. 2).

Critical strengths
  • Interesting approach to switch out agentic tool-calls with a model which one can actually optimize for the problem.
  • The approach makes logically sense and seems a reasonable thing to do.
  • Results show clear gains w.r.t. the baseline.
Critical weaknesses

The paper writing can be improved. Most critically, L372-L377 moves the most relevant experiment for this paper to the supplementary (!) (see C.3 Table 13). This experiment applies direct RL to the problem to adapt the weights for the specific benchmarks this paper addresses. Obviously this is the main baseline to beat since otherwise you are comparing a finetuned method to zero-shot performance. I first thought that this experiment was absent which would be a strong reason for rejecting this paper. It makes no sense to hide this experiment in the supplementary material. The comparison also convincingly demonstrates that the proposed method is better. So please put this in the main paper INSIDE table 1 (other rows can be easily moved to the supplementary; like all 'general models w/o tools').

Other:

  • 3.1: \pi_b is talked about as model, yet these are probably the parameters. This symbol is also absent from Fig 2 so using this symbol is a bit confusing here.
  • Generally, V and PC have no connections to the names of the specialists. Consider renaming (see L195 and L245). In L245, G is undefined. Is this the base model? L248: E_g is undefined. Is this the 'direct' role?
  • It is also not totally clear for the reader why the base model remains frozen and why it is called sometimes directly. Calling it directly seems an odd design choice. The frozen parameters are likely for LoRA but it would help to make this explicit.
  • Dual LoRA is not clearly defined when reading the results section.
  • Why is EgoP2C + additional SFT not the main method?
  • It would probably be easier for the reader to show first Table 3 expanded with the SenseNova and Qwen baselines. Then afterwards show the results of the main method in context of other LVLMs (e.g. Table 1, although I still wonder whether all rows are really useful for most readers).
  • Many tables miss averages. Please add them. Much easier to understand differences by comparing single numbers.
Questions for authors

Why would it ever be a good idea for the planner to call the frozen base model directly? This seems an odd design choice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.