- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper seeks to reconstruct a spatio-temporal field from observations from a fixed trajectory. The key problem addressed is how to adapt the reconstruction when the deployment trajectory is different from the training trajectory. The paper introduces a trajectory translator (TT) which maps the deployment trajectory to a trajectory in the original space to guide the reconstruction. The results are empirically evaluated in a number of domains, and are shown to improve over No TT case and the ControlNet baseline in most cases.
Critical strengths
The paper addresses an interesting problem of field reconstruction when the training and deployment trajectories are different.
Multiple reconstructors and multiple domains are evaluated.
The empirical results are relatively strong. TT outperforms ControlNet in 14 out of 18 cases (reconstructorsXdomains).
The paper also makes some relevant theoretical claims with accompanying proofs.
Critical weaknesses
The paper is poorly written and hard to understand. Figure 1 has a lot of information but has not been carefully narrated in the text.
A lot of terms remain unexplained, eg, DF_A in Proposition 1, Dagger iteration, etc. It seems LLMs are overused from hypothesis generation to data cleaning to editing. Not enough attention was paid to explaining the main ideas in a human-digestable form. The length of the paper and the appendices, and lack of sufficient narrative explanations make the paper hard to understand.
The main technical weakness is that the deployment does not work in a zero-shot fashion. Instead the teacher generates a set of observation and optimal target pairs using privileged ground truth of the field, which are fed to the student learner that learns to map deployment observations to appropriate targets. This is a lot to ask and reduces the significance of the paper.
Post-rebuttal comments: Thanks for the clarifications and the new experiments. They significantly improve the paper. I am happy to raise my score.
Questions for authors
The teacher seems to assume the privileged information of the ground truth field. Isn't this hard to provide in more realistic settings? Line 171. It is not clear what the ideal target for z is if not a. This needs more clear explanation at or before this point. Line 182. Whit is an admissible action set? Line 183. Very little is said about ControlNet in the main paper. How does it differ from your approach? Are there other relevant baselines to compare against?
4 comments
Response to Reviewer CiqA (2/4)
Comment 2 (W2, Q1): Full-field supervision and deployment requirements
The reviewer raises two related concerns: TT requires adaptation rather than arbitrary zero-shot deployment, and its teacher uses privileged full-field information that may be difficult to obtain in practice.
The distinction between training and deployment is explicitly stated in Section 2.3 and the caption of Figure 1. The full field is used during offline training to optimize teacher targets. At deployment, the student receives the sensing configuration and its measurements , and the frozen reconstructor processes the predicted virtual observations:
\qquad \widehat X=F_{\mathcal A}(z).$$ Neither the full field of the deployment instance nor teacher optimization is required at deployment. Here, $\mathcal B$ denotes the sensing trajectory, $b$ its sparse measurements, and $X$ the full physical field. The practical training requirement is a representative offline dataset of full fields. Given such a dataset, observations for a target trajectory $\mathcal B$ can be constructed through its observation operator. The same offline fields can therefore support adaptation to different sensing configurations, without collecting a new full-field measurement for every deployment trajectory. Our current experiments use simulated fields. For an application, suitable sources could include archived dense measurements or representative simulations, with their suitability assessed against the intended deployment conditions. Access to such offline supervision is an assumption of the present method. Regarding zero-shot deployment, the paper focuses on configuration-to-configuration adaptation of a frozen reconstructor using offline training. The limitations paragraph in the Conclusion explicitly states that arbitrary-configuration zero-shot adaptation remains outside the established scope. The practical benefit is that a change in sensing configuration can be handled through a learned input translator while retaining the pretrained reconstruction model. We have clarified the offline training requirement in the Figure 1 caption.Response to Reviewer CiqA (3/4)
Comment 3 (W1, Q2): Why virtual observations are not trained to reproduce genuine reference measurements
The reviewer asks what the target for should be if it is not the genuine reference observation (line 171).
This distinction is stated in Section 2.2, which defines virtual observations as control inputs to the frozen reconstructor. Our objective is to reconstruct the physical field from the evidence collected along the deployment trajectory . We therefore seek virtual observations that convey this evidence through the familiar reference interface . The desired target is an input that enables accurate reconstruction from ; it need not reproduce the measurements that would have been collected on .
The adaptive-control motivation is introduced in Section 2.2, and the connection to the MIT rule is developed in Appendix B.2. The MIT rule adjusts control parameters according to the output error and its sensitivity to those parameters. Following this principle, we treat the virtual observations as optimizable input variables in the computation graph, while keeping the reconstructor parameters fixed. As described in Section 2.3, the teacher uses constrained Adam updates to optimize
\ell_F\left(F_{\mathcal A}(z),X\right).$$ The resulting feasible teacher inputs provide supervision for the student, which learns to predict them from deployment-accessible observations. Proposition 1 provides a local justification for why adjusted virtual observations can outperform genuine reference observations. Taking $z_0=a$, under the stated differentiability and feasibility assumptions and when $J^\top e\neq0$, a sufficiently small feasible input adjustment yields $$\lVert F_{\mathcal A}(z_\eta)-X\rVert_2^2 < \lVert F_{\mathcal A}(a)-X\rVert_2^2.$$ Thus, even the genuine reference measurements need not minimize reconstruction error for a fixed, imperfect reconstructor. Optimizing the input provides a way to compensate for part of that error. Predicting $a$ is also explicitly evaluated in the manuscript. The A-supervised comparison in Appendix F.2.1 trains a translator to predict genuine reference measurements from deployment observations. Under the reported training recipes, TT achieves lower reconstruction error in 16 of the 18 settings. For example, on Plate Transformer, A-supervised obtains 0.5804, compared with 0.1538 for TT. This comparison supports evaluating virtual observations by the reconstruction they induce. We will make the connections among this comparison, Proposition 1, and the target definition in Section 2.2 more explicit.Response to Reviewer CiqA (4/4)
Comment 4 (Q4, Q5): Comparison with ControlNet and additional adaptation baselines
The reviewer asks how ControlNet differs from TT and whether other relevant baselines can be included. We agree that the main-text explanation of these comparisons should be more detailed.
The key distinction is where adaptation acts within the reconstruction pipeline. These methods expose different quantities for control:
- Fine-tuning updates the reconstructor’s parameters; LoRA introduces trainable low-rank changes to its effective weights.
- ControlNet-style adaptation introduces observation-conditioned residual corrections to internal features.
- FiLM modulates intermediate features through conditional scaling and shifting.
- TT adjusts the virtual observations supplied to the original input interface, keeping the reconstructor’s parameters and internal architecture fixed.
The ControlNet implementations are described in Appendix G.4. The main-table comparison uses conditional residual adapters whose trainable parameter counts match TT within 0.3%. These adapters inject corrections into internal representations through zero-initialized output projections. TT instead applies its learned correction at the observation input. We will summarize this distinction in the main text and provide a direct reference to the implementation details.
To extend the comparison, we evaluated LoRA and FiLM on four Transformer settings. We also include E2E-TT, which trains the translator directly through the frozen reconstructor’s field loss and provides another training variant within the same input-interface adaptation framework.
| Setting | TS-TT | ControlNet | E2E-TT | FiLM | LoRA |
|---|---|---|---|---|---|
| NS–Uniform Transformer | 0.2312 | 0.3105 | 0.2208 | 0.2377 | 0.2384 |
| NS–Scan Transformer | 0.3982 | 0.3676 | 0.3162 | 0.2760 | 0.2900 |
| Airfoil Transformer | 0.004030 | 0.002795 | 0.002394 | 0.002775 | 0.002933 |
| Plate Transformer | 0.1538 | 0.3857 | 0.0694 | 0.0778 | 0.2361 |
Viewed from the control perspective, these comparisons examine adaptation through effective weights, intermediate features, and observation inputs. TT specifically studies the control capability of the input interface under observation-interface shifts. This also offers a practical advantage when access to model internals is restricted: training requires forward evaluations and derivatives with respect to the input, but no permission to modify reconstructor weights or insert controls into internal layers. Deployment requires only forward evaluation of the frozen reconstructor. We have stated these access requirements explicitly alongside the expanded baseline discussion.
Response to Reviewer CiqA (1/4)
We thank the reviewer for recognizing the relevance of the problem, the strong empirical results, and the accompanying theoretical analysis. Below, we address the specific concerns about presentation, deployment requirements, the virtual-observation target, and comparisons with alternative adaptation methods. We have expanded the Figure 1 caption and clarified the notation, DAgger procedure, and comparisons with alternative adaptation methods. We will add the complete expanded baseline comparison after completing the remaining settings.
Comment 1 (W1, Q3): Clarification of notation and the training procedure
The reviewer raises concerns about the derivative notation in Proposition 1, the meaning of the admissible input set, the DAgger iterations, and the explanation accompanying Figure 1. We address each point below, referring to the corresponding formulation in the manuscript.
The reviewer identifies DFA(z0) in Proposition 1 (line 179) as unexplained notation. The proposition explicitly assumes that FA is continuously differentiable in a neighborhood of z0. We use the conventional notation D for the derivative, so that
=\left.\frac{\partial F_{\mathcal A}(z)}{\partial z}\right|_{z=z_0}.$$ The relationship $F_{\mathcal A}(z):=R_\theta(\mathcal A,z)$ is defined in Section 2.1. Our use of $F_{\mathcal A}$ makes the input–output mapping explicit, with the reference interface $\mathcal A$ and reconstructor parameters $\theta$ held fixed. Proposition 1 develops the local input-compensation argument motivated by adaptive control and the MIT rule, discussed in Section 2.2 and Appendix B.2. This formulation allows the argument to apply independently of the reconstructor’s particular architecture. We will explicitly identify $DF_{\mathcal A}(z_0)$ as the input Jacobian at its first occurrence. The reviewer asks about the admissible set in the assumption at line 182. The admissible input set $\mathcal Z$ is the set of virtual observations permitted by the optimization constraints. The condition $z_0\in\operatorname{int}(\mathcal Z)$ ensures that a sufficiently small gradient step remains feasible. Together with $J^\top e\neq0$, where $e=F_{\mathcal A}(z_0)-X$, and the stated differentiability assumption, this guarantees that the update $$z_\eta=z_0-\eta J^\top e$$ reduces reconstruction error for sufficiently small $\eta>0$ while staying within $\mathcal Z$. In our bounded-residual implementations, this restriction is implemented by constraining virtual inputs around an initialization computed from deployment observations. These bounds regularize the input adjustment and prevent it from moving arbitrarily far from the initialization. We will make this connection between the proposition’s feasibility assumption and the implementation explicit. The reviewer requests clarification of the DAgger iterations. Inspired by Learning Using Privileged Information (LUPI), we use full-field information available during training to construct teacher supervision, while the student receives only deployment-accessible inputs. DAgger refers to Dataset Aggregation. In settings using our DAgger-style mechanism, teacher-generated corrections are added to the supervision dataset, and the student is further trained on the aggregated targets. The DAgger-style implementation was described in Appendix G.3 of the original manuscript; we have now added a brief explanation to Section 2.3.1 of the revised manuscript. The reviewer notes that Figure 1 contains substantial information without sufficient accompanying narration. Figure 1 provides a compact overview of the design developed throughout Section 2: Section 2.1 introduces the problem formulation; Section 2.2 explains the virtual-observation target and its control interpretation; Section 2.3 presents the teacher–student framework; and Section 2.4 describes the offline and online formulations. The components and training flow summarized in Figure 1 are developed throughout Section 2. We have expanded the Figure 1 caption