acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Translator for Amortized Field Reconstructors under Observation-Interface Shift

Abstract

Amortized field reconstruction models learn to infer complete physical fields from partial observations under the sensing configurations represented during training. When training is restricted to a fixed sensing trajectory, the learned reconstruction map may become specialized to its observation interface. At deployment, however, novel or adaptively selected trajectories can introduce observation-interface distribution shift, constituting an out-of-distribution (OOD) setting. We introduce Trajectory Translator (TT), an add-on module that maps deployment observations to virtual observations on the original observation interface, guiding field reconstruction without modifying the frozen reconstructor. Under a finite-state Bayesian formulation, we characterize when bounded virtual observations preserve deployment evidence exactly. For deterministic reconstructors, we relate squared reconstruction risk to approximation of the conditional mean and evaluate TT through empirical reconstruction errors. Experiments across physical-field benchmarks show that TT improves reconstruction under the evaluated observation-interface shifts while keeping the underlying reconstructors frozen, reducing relative reconstruction error by up to 88.9% for the Transformer on the plate-strain benchmark. These results demonstrate that virtual observations on the original interface can effectively steer a frozen reconstructor toward more accurate field estimates. At deployment, TT translates each observation sequence or block in a single forward pass. In the measured offline settings, TT adds only 0.39–0.57 ms per complete sequence, introducing little additional computational overhead. Code is available at https://anonymous.4open.science/r/Trajectory-Translator-3D38/.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physical field reconstruction, Amortized inference, Observation-interface shift

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CiqAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

The paper seeks to reconstruct a spatio-temporal field from observations from a fixed trajectory. The key problem addressed is how to adapt the reconstruction when the deployment trajectory is different from the training trajectory. The paper introduces a trajectory translator (TT) which maps the deployment trajectory to a trajectory in the original space to guide the reconstruction. The results are empirically evaluated in a number of domains, and are shown to improve over No TT case and the ControlNet baseline in most cases.

Critical strengths

The paper addresses an interesting problem of field reconstruction when the training and deployment trajectories are different.

Multiple reconstructors and multiple domains are evaluated.

The empirical results are relatively strong. TT outperforms ControlNet in 14 out of 18 cases (reconstructorsXdomains).

The paper also makes some relevant theoretical claims with accompanying proofs.

Critical weaknesses

The paper is poorly written and hard to understand. Figure 1 has a lot of information but has not been carefully narrated in the text.
A lot of terms remain unexplained, eg, DF_A in Proposition 1, Dagger iteration, etc. It seems LLMs are overused from hypothesis generation to data cleaning to editing. Not enough attention was paid to explaining the main ideas in a human-digestable form. The length of the paper and the appendices, and lack of sufficient narrative explanations make the paper hard to understand.

The main technical weakness is that the deployment does not work in a zero-shot fashion. Instead the teacher generates a set of observation and optimal target pairs using privileged ground truth of the field, which are fed to the student learner that learns to map deployment observations to appropriate targets. This is a lot to ask and reduces the significance of the paper.

Post-rebuttal comments: Thanks for the clarifications and the new experiments. They significantly improve the paper. I am happy to raise my score.

Questions for authors

The teacher seems to assume the privileged information of the ground truth field. Isn't this hard to provide in more realistic settings? Line 171. It is not clear what the ideal target for z is if not a. This needs more clear explanation at or before this point. Line 182. Whit is an admissible action set? Line 183. Very little is said about ControlNet in the main paper. How does it differ from your approach? Are there other relevant baselines to compare against?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CiqA (1/4)

We thank the reviewer for recognizing the relevance of the problem, the strong empirical results, and the accompanying theoretical analysis. Below, we address the specific concerns about presentation, deployment requirements, the virtual-observation target, and comparisons with alternative adaptation methods. We have expanded the Figure 1 caption and clarified the notation, DAgger procedure, and comparisons with alternative adaptation methods. We will add the complete expanded baseline comparison after completing the remaining settings.

Comment 1 (W1, Q3): Clarification of notation and the training procedure

The reviewer raises concerns about the derivative notation in Proposition 1, the meaning of the admissible input set, the DAgger iterations, and the explanation accompanying Figure 1. We address each point below, referring to the corresponding formulation in the manuscript.

The reviewer identifies DFA(z0)DF_{\mathcal A}(z_0) in Proposition 1 (line 179) as unexplained notation. The proposition explicitly assumes that FAF_{\mathcal A} is continuously differentiable in a neighborhood of z0z_0. We use the conventional notation DD for the derivative, so that

=\left.\frac{\partial F_{\mathcal A}(z)}{\partial z}\right|_{z=z_0}.$$ The relationship $F_{\mathcal A}(z):=R_\theta(\mathcal A,z)$ is defined in Section 2.1. Our use of $F_{\mathcal A}$ makes the input–output mapping explicit, with the reference interface $\mathcal A$ and reconstructor parameters $\theta$ held fixed. Proposition 1 develops the local input-compensation argument motivated by adaptive control and the MIT rule, discussed in Section 2.2 and Appendix B.2. This formulation allows the argument to apply independently of the reconstructor’s particular architecture. We will explicitly identify $DF_{\mathcal A}(z_0)$ as the input Jacobian at its first occurrence. The reviewer asks about the admissible set in the assumption at line 182. The admissible input set $\mathcal Z$ is the set of virtual observations permitted by the optimization constraints. The condition $z_0\in\operatorname{int}(\mathcal Z)$ ensures that a sufficiently small gradient step remains feasible. Together with $J^\top e\neq0$, where $e=F_{\mathcal A}(z_0)-X$, and the stated differentiability assumption, this guarantees that the update $$z_\eta=z_0-\eta J^\top e$$ reduces reconstruction error for sufficiently small $\eta>0$ while staying within $\mathcal Z$. In our bounded-residual implementations, this restriction is implemented by constraining virtual inputs around an initialization computed from deployment observations. These bounds regularize the input adjustment and prevent it from moving arbitrarily far from the initialization. We will make this connection between the proposition’s feasibility assumption and the implementation explicit. The reviewer requests clarification of the DAgger iterations. Inspired by Learning Using Privileged Information (LUPI), we use full-field information available during training to construct teacher supervision, while the student receives only deployment-accessible inputs. DAgger refers to Dataset Aggregation. In settings using our DAgger-style mechanism, teacher-generated corrections are added to the supervision dataset, and the student is further trained on the aggregated targets. The DAgger-style implementation was described in Appendix G.3 of the original manuscript; we have now added a brief explanation to Section 2.3.1 of the revised manuscript. The reviewer notes that Figure 1 contains substantial information without sufficient accompanying narration. Figure 1 provides a compact overview of the design developed throughout Section 2: Section 2.1 introduces the problem formulation; Section 2.2 explains the virtual-observation target and its control interpretation; Section 2.3 presents the teacher–student framework; and Section 2.4 describes the offline and online formulations. The components and training flow summarized in Figure 1 are developed throughout Section 2. We have expanded the Figure 1 caption
AuthorsOpenReview ↗

Response to Reviewer CiqA (2/4)

Comment 2 (W2, Q1): Full-field supervision and deployment requirements

The reviewer raises two related concerns: TT requires adaptation rather than arbitrary zero-shot deployment, and its teacher uses privileged full-field information that may be difficult to obtain in practice.

The distinction between training and deployment is explicitly stated in Section 2.3 and the caption of Figure 1. The full field XX is used during offline training to optimize teacher targets. At deployment, the student receives the sensing configuration B\mathcal B and its measurements bb, and the frozen reconstructor processes the predicted virtual observations:

\qquad \widehat X=F_{\mathcal A}(z).$$ Neither the full field of the deployment instance nor teacher optimization is required at deployment. Here, $\mathcal B$ denotes the sensing trajectory, $b$ its sparse measurements, and $X$ the full physical field. The practical training requirement is a representative offline dataset of full fields. Given such a dataset, observations for a target trajectory $\mathcal B$ can be constructed through its observation operator. The same offline fields can therefore support adaptation to different sensing configurations, without collecting a new full-field measurement for every deployment trajectory. Our current experiments use simulated fields. For an application, suitable sources could include archived dense measurements or representative simulations, with their suitability assessed against the intended deployment conditions. Access to such offline supervision is an assumption of the present method. Regarding zero-shot deployment, the paper focuses on configuration-to-configuration adaptation of a frozen reconstructor using offline training. The limitations paragraph in the Conclusion explicitly states that arbitrary-configuration zero-shot adaptation remains outside the established scope. The practical benefit is that a change in sensing configuration can be handled through a learned input translator while retaining the pretrained reconstruction model. We have clarified the offline training requirement in the Figure 1 caption.
AuthorsOpenReview ↗

Response to Reviewer CiqA (3/4)

Comment 3 (W1, Q2): Why virtual observations are not trained to reproduce genuine reference measurements

The reviewer asks what the target for zz should be if it is not the genuine reference observation aa (line 171).

This distinction is stated in Section 2.2, which defines virtual observations as control inputs to the frozen reconstructor. Our objective is to reconstruct the physical field from the evidence collected along the deployment trajectory B\mathcal B. We therefore seek virtual observations that convey this evidence through the familiar reference interface A\mathcal A. The desired target is an input that enables accurate reconstruction from (B,b)(\mathcal B,b); it need not reproduce the measurements that would have been collected on A\mathcal A.

The adaptive-control motivation is introduced in Section 2.2, and the connection to the MIT rule is developed in Appendix B.2. The MIT rule adjusts control parameters according to the output error and its sensitivity to those parameters. Following this principle, we treat the virtual observations zz as optimizable input variables in the computation graph, while keeping the reconstructor parameters fixed. As described in Section 2.3, the teacher uses constrained Adam updates to optimize

\ell_F\left(F_{\mathcal A}(z),X\right).$$ The resulting feasible teacher inputs provide supervision for the student, which learns to predict them from deployment-accessible observations. Proposition 1 provides a local justification for why adjusted virtual observations can outperform genuine reference observations. Taking $z_0=a$, under the stated differentiability and feasibility assumptions and when $J^\top e\neq0$, a sufficiently small feasible input adjustment yields $$\lVert F_{\mathcal A}(z_\eta)-X\rVert_2^2 < \lVert F_{\mathcal A}(a)-X\rVert_2^2.$$ Thus, even the genuine reference measurements need not minimize reconstruction error for a fixed, imperfect reconstructor. Optimizing the input provides a way to compensate for part of that error. Predicting $a$ is also explicitly evaluated in the manuscript. The A-supervised comparison in Appendix F.2.1 trains a translator to predict genuine reference measurements from deployment observations. Under the reported training recipes, TT achieves lower reconstruction error in 16 of the 18 settings. For example, on Plate Transformer, A-supervised obtains 0.5804, compared with 0.1538 for TT. This comparison supports evaluating virtual observations by the reconstruction they induce. We will make the connections among this comparison, Proposition 1, and the target definition in Section 2.2 more explicit.
AuthorsOpenReview ↗

Response to Reviewer CiqA (4/4)

Comment 4 (Q4, Q5): Comparison with ControlNet and additional adaptation baselines

The reviewer asks how ControlNet differs from TT and whether other relevant baselines can be included. We agree that the main-text explanation of these comparisons should be more detailed.

The key distinction is where adaptation acts within the reconstruction pipeline. These methods expose different quantities for control:

  • Fine-tuning updates the reconstructor’s parameters; LoRA introduces trainable low-rank changes to its effective weights.
  • ControlNet-style adaptation introduces observation-conditioned residual corrections to internal features.
  • FiLM modulates intermediate features through conditional scaling and shifting.
  • TT adjusts the virtual observations supplied to the original input interface, keeping the reconstructor’s parameters and internal architecture fixed.

The ControlNet implementations are described in Appendix G.4. The main-table comparison uses conditional residual adapters whose trainable parameter counts match TT within 0.3%. These adapters inject corrections into internal representations through zero-initialized output projections. TT instead applies its learned correction at the observation input. We will summarize this distinction in the main text and provide a direct reference to the implementation details.

To extend the comparison, we evaluated LoRA and FiLM on four Transformer settings. We also include E2E-TT, which trains the translator directly through the frozen reconstructor’s field loss and provides another training variant within the same input-interface adaptation framework.

SettingTS-TTControlNetE2E-TTFiLMLoRA
NS–Uniform Transformer0.23120.31050.22080.23770.2384
NS–Scan Transformer0.39820.36760.31620.27600.2900
Airfoil Transformer0.0040300.0027950.0023940.0027750.002933
Plate Transformer0.15380.38570.06940.07780.2361

Viewed from the control perspective, these comparisons examine adaptation through effective weights, intermediate features, and observation inputs. TT specifically studies the control capability of the input interface under observation-interface shifts. This also offers a practical advantage when access to model internals is restricted: training requires forward evaluations and derivatives with respect to the input, but no permission to modify reconstructor weights or insert controls into internal layers. Deployment requires only forward evaluation of the frozen reconstructor. We have stated these access requirements explicitly alongside the expanded baseline discussion.

Reviewer_2s5NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to reuse frozen physical-field reconstructors when sensor layouts or trajectories change. The proposed Trajectory Translator (TT) maps deployment observations to virtual observations on the original interface. During training, a teacher optimizes these virtual inputs using full-field ground truth, and a GRU student learns to predict them for efficient deployment. The central idea is to evaluate virtual observations by their reconstruction utility rather than their fidelity to genuine measurements on the original interface. Results across multiple physical tasks and reconstructor architectures support the approach’s effectiveness. Overall, the paper presents a practically useful adaptation method with a clearly defined scope, and I lean toward acceptance.

Critical strengths
  • A clear and practically relevant problem. The paper addresses performance degradation when an existing reconstructor encounters a different observation interface. It recognizes that reconstruction errors may reflect either insufficient observational information or the model’s inability to exploit new observations. This provides a clear motivation for adapting pretrained or deployed reconstructors.

  • An insightful formulation of virtual observations. TT uses the original observation interface to steer a frozen reconstructor, evaluating virtual inputs by the quality of the resulting reconstruction. This allows input adjustments to compensate for the reconstructor’s own errors and provides a common adaptation mechanism across different architectures.

  • Broad experimental coverage with complementary metrics. Experiments cover four physical datasets, multiple sensing configurations, and several reconstructor architectures. In Table 1, TT improves over the unadapted models in all 18 reported settings and outperforms parameter-matched adapters in 14 settings. Additional MSE, deployment-observation consistency, and runtime analyses support its effectiveness and deployment feasibility.

  • Useful clarification of theoretical boundaries. The finite-state Bayesian analysis distinguishes limitations arising from interface expressivity and bounded input adjustments. The appendices also distinguish conditional-mean matching from full-posterior matching, and teacher-input regression from output-risk minimization. These distinctions clarify what the theory guarantees and what remains to be established empirically for neural reconstructors.

Critical weaknesses
  • The additional benefit of the teacher–student procedure could be clarified further. Appendix E.3 considers joint label-and-field-loss fine-tuning in the toy experiment, but the advantage of distillation over directly training the student remains unclear for the physical tasks. In one representative setting, the authors could retain the student architecture, available observations, training data, and virtual-input constraints, while training the translator directly through the frozen reconstructor’s full-field loss. Gradients would pass through the reconstructor, updating only the translator, without teacher-generated labels. Comparing reconstruction accuracy, stability, and total training cost—including teacher-label generation—would help establish the value of the two-stage procedure.

  • The sources of improvement over some baselines in the offline setting could be analyzed more clearly. Section 2.4.1 and Appendix G.3 specify a bidirectional GRU, allowing the virtual observation at time (t) to encode later measurements. This information can influence the output even when the downstream reconstructor uses causal attention or observation prefixes. In contrast, some No TT baselines in Appendix G.1 access only current and past observations. These comparisons therefore leave unclear how much improvement comes from interface adaptation and how much comes from access to future observations. Although the design is valid for offline reconstruction, a representative temporal experiment comparing No TT, a causal TT trained and evaluated using observation prefixes, and the original offline TT with the same frozen reconstructor would show how much adaptation benefit persists under matched observation availability. This concern is limited to comparisons with different information access, since offline ControlNet also receives complete sequences.

  • The scope and stability of unseen-trajectory generalization could be presented more clearly. Figure 4(d) provides encouraging evidence, but primarily evaluates NS–Uniform with a frozen Transformer. The authors could specify how seen trajectories, unseen trajectories within seen families, and unseen trajectory families are generated, including their variation ranges, counts, and splits. Reporting means and standard deviations over a few independent student-training seeds for this existing experiment would help readers assess the extent of interface variation being tested and the stability of the gains. These additions would strengthen the current evidence without requiring arbitrary-configuration generalization or a new active-sensing task.

Questions for authors

Please refer to weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2s5N (1/4)

We thank the reviewer for the careful assessment and for recognizing the practical relevance of the problem, the formulation of virtual observations, and the experimental and theoretical contributions. We appreciate the concrete suggestions for clarifying the teacher–student procedure, separating the effects of interface adaptation and access to future observations, and specifying the scope of trajectory generalization. We address these points below.

We have clarified the two training variants and added accuracy–cost comparisons, causal-GRU ablations, and detailed cross-trajectory protocols with three-seed results. We will add E2E-TT results for the remaining benchmark settings.

AuthorsOpenReview ↗

Response to Reviewer 2s5N (2/4)

Comment 1 (W1): Benefits of the teacher–student procedure and comparison with end-to-end training

The reviewer asks what the teacher–student procedure offers over directly training the translator through the frozen reconstructor’s field loss. We agree that these benefits should be explained more clearly and have added the suggested comparison. The end-to-end formulation suggested by the reviewer is also consistent with our central idea of adapting a frozen reconstructor by adjusting virtual observations through its input interface.We refer to these two training variants as TS-TT and E2E-TT, respectively. Our motivation for adopting the teacher–student procedure has two main aspects.

First, TS-TT provides an explicit diagnostic of the input interface’s control capability. The decomposition helps distinguish two questions: whether effective virtual inputs can be found for a given instance, and whether a student can learn to predict such inputs from deployment-accessible observations. When the teacher finds a virtual input with low reconstruction error, it demonstrates, for that instance, how effectively the frozen reconstructor can be controlled through its input. The student then learns a deployable mapping, and its performance can be assessed against this teacher reference.

Second, TS-TT makes the teacher’s optimization results reusable. The optimized virtual inputs can be stored as a supervision dataset for learning

(B,b)⟶z⋆.(\mathcal B,b)\longrightarrow z^{\star}.

Once this dataset is fixed, different student architectures, hidden dimensions, and optimization settings can be explored using the same cached targets. With cached-label supervision, subsequent student training does not require repeated backpropagation through the frozen reconstructor. In contrast, each E2E-TT training run optimizes the translator through the reconstructor’s field loss. TS-TT therefore allows the teacher-generation cost to be shared across multiple student configurations, making supervision reuse particularly valuable during architecture selection and hyperparameter tuning.

To quantify this reuse benefit, we analyzed a fixed-cache sweep over five student configurations on Plate Transformer, each trained for 40,000 updates. The cumulative costs, including the initial teacher-generation cost, are estimated as follows:

Number of student configurationsTS-TT cost (H800 GPU-hours)E2E-TT cost(H800 GPU-hours)E2E/TS cost ratio
19.008.840.98×
29.0417.711.96×
39.0726.562.93×
49.1235.423.89×
59.1544.284.84×

As more student configurations are explored, the teacher-generation cost is amortized across these runs. Across five configurations, the projected cumulative cost is 9.15 H800 GPU-hours for TS-TT, compared with 44.28 H800 GPU-hours for E2E-TT.

Following the reviewer’s suggestion, we also evaluated E2E-TT across six physical benchmark settings. In these experiments, the translator is trained directly through the frozen reconstructor’s field loss, without teacher-generated labels. The reconstruction results are:

SettingTT, original main-table recipeE2E-TT
NS–Uniform Transformer0.23120.2208
NS–Scan FNO0.34490.3286
NS–Scan Mobile-SHRED0.37550.3199
Plate Transformer0.15380.0694
Pollutant SHRED0.23030.2226
Pollutant FNO0.24270.2621

Lower is better. Each setting retains its original evaluation metric and partition. The TT column reports the original training recipes; these comparisons are separate from the fixed-cache cost analysis above.

We additionally conducted a controlled NS–Uniform Transformer comparison using 256 training fields, 96 held-out development fields, and three independent student-training seeds. Within each seed, the methods share the student initialization, training data, observation inputs, virtual-input constraints, and an 8,000-update budget. The reconstruction errors, reported as mean ± standard deviation across seeds, are 0.32660 ± 0.00237 for label-supervised TS-TT and 0.31078 ± 0.00207 for E2E-TT.

The core contribution of TT is to demonstrate the control capability of the observation input interface for adapting a frozen reconstructor under observation-interface shifts. Both the teacher–student procedure and direct end-to-end training implement this input-based adaptation. We will therefore present E2E-TT as another variant of TT, alongside TS-TT, and explain their respective benefits in reconstruction accuracy, interface diagnostics, and supervision reuse.

AuthorsOpenReview ↗

Response to Reviewer 2s5N (3/4)

Comment 2 (W2): Future observations in offline TT and causal ablation

The reviewer asks how much of the offline improvement comes from interface adaptation and how much comes from access to future observations. We agree that a bidirectional GRU allows the virtual input at time tt to encode later measurements, even when the downstream reconstructor is causal. Offline TT is designed for full-sequence reconstruction. To quantify the adaptation benefit under matched causal observation access, we trained and evaluated unidirectional GRU translators whose output at time tt depends only on the observation prefix up to tt. These variants reuse the original teacher labels and frozen reconstructors, while retaining the corresponding student training objectives and update budgets.

SettingNo TTBidirectional GRU TTUnidirectional GRU TT
NS–Uniform Transformer0.44160.23120.2405
NS–Scan Transformer0.61850.37350.4196
Airfoil Transformer (×10⁻²)2.13500.40300.4080
Airfoil SHRED (×10⁻²)2.14800.50700.4900

The causal variants retain at least 81% of the reconstruction-error reduction achieved by the bidirectional variants relative to No TT in all four settings. Restricting access to observation prefixes leads to modest increases in reconstruction error in the first three settings, while Airfoil SHRED improves slightly. These results show that substantial adaptation benefits persist under causal observation access.

The information-access conditions are specified in Section 2.4. Offline TT uses complete observation sequences. Online TT respects observation availability at each block update: it generates virtual observations after the current block has been collected, uses no observations from later blocks, and leaves previously submitted virtual observations unchanged.

Reviewer_s1zQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important trajectory based reconstruction problem to recover physical fields which is not easy to solve. A reconstructor is trained to reconstruct a physical field X from sensor readings along a trajectory A. Given a new trajectory B, directly feeding B's readings to the trained reconstructor can dramatically increase the reconstruction error. Plus, retraining on B can be compute-heavy. To mitigate this error, the paper proposes Trajectory Translator (TT), which is a teacher-student framework that translates B's readings into virtual readings on A for the reconstructor. It demonstrates lower recontruction error on majority settings over all baselines including ControlNet on 4 simulated physical datasets with 4 reconstructors.

Critical strengths

Problem & motivation: The research problem is clear and practical. Figure 2 and 3 and a specific session for motivation shows the question clearly, that reconstruction error largely increases due to the changes of sensing trajectory from A to B.

Method / novelty: It is new to use teacher-student framework on sensing-trajectory shift problem.

Evidence: It shows that TT beats no TT and interpolation, and field refinement in almost all settings, and beats ControlNet in 14 out of 18 settings.

Significance: This is an efficient add-on method that could be efficiently trained and used on pre-trained field reconstructors, and the method works well cross 4 different architectures of field reconstructor.

Critical weaknesses

a. The advantage of avoiding retraining is not fully clear. The paper only reports inference time in table 2, but it doesn't mention their method's specific training time + gpu hours + hardware for training as well as comparison with retraining on B.

b. The generalization evidence is limited. The paper shows that the method needs to train on every pair of layout B and reconstructor. There is one cross-trajectory test which uses NS-Uniform with Transformer as reconstructor but its gain on unseen trajectories is much lower.

Questions for authors

The paper use simulated data for all training and evaluation with no real sensor data. This might be a default in this field and all baselines do share the same assumption. The method's training needs the full field on B, so I'm curious in practice, how can Trajectory Translator be trained in a real deployment? is it feasible to get the full field of B?

Why not fine-tune the pre-trained reconstructor on B directly? the paper only mentioned retraining could be compute-heavy.

why split into teacher and student? the student can be trained end-to-end as well from b to X. Is this considered in ablation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer s1zQ (1/4)

We thank the reviewer for recognizing the practical significance of the problem and the effectiveness of TT across different reconstructor architectures. We also appreciate the reviewer’s constructive questions about practical deployment and alternative adaptation strategies, including fine-tuning, retraining, and end-to-end TT. We address these concerns below.

We have revised the adaptation motivation and TS-TT/E2E-TT description, and added training-cost comparisons and cross-trajectory stability results. We will add E2E-TT results for the remaining benchmark settings.

AuthorsOpenReview ↗

Response to Reviewer s1zQ (2/4)

Comment 1 (Q3): Why use a teacher–student procedure rather than end-to-end training?

The reviewer asks whether the student can instead be trained end to end from deployment observations to the reconstructed field. We agree that the original manuscript could explain the benefits of the teacher–student design more clearly. End-to-end training is indeed a viable alternative, and we have now evaluated it in additional experiments. Our motivation for the two-stage design is twofold.

First, the teacher–student design separates interface controllability from learning deployable input adaptation. TT treats the observation input as a control interface for addressing observation shift in a frozen reconstructor. This raises two distinct questions: what reconstruction quality can be achieved through this interface, and whether effective inputs can be predicted from the available observations. The teacher uses full-field training targets to optimize virtual observations, providing an empirical diagnostic of interface controllability. The student then learns to predict these virtual observations from (B,b)(\mathcal B,b), assessing their deployability without access to full-field ground truth at inference. The two-stage design therefore allows these questions to be examined separately.

Second, the teacher–student design makes expensive input-optimization results reusable. The optimized virtual observations are cached as supervision for subsequent student training. With the reconstructor, training data, and target-generation protocol fixed, the same teacher targets can support multiple student architectures or hyperparameter configurations. Label-supervised student training then only fits the mapping (B,b)→z⋆(\mathcal B,b)\rightarrow z^\star, avoiding repeated backpropagation through the reconstructor. By comparison, each independently trained E2E-TT configuration optimizes its translator through the frozen reconstructor using the field loss. This reuse is particularly useful when exploring different student configurations.

We quantified this benefit through a fixed-cache cost projection on Plate Transformer, using 5,000 training fields and 40,000 student updates per configuration. The one-time cost of generating teacher targets with 128 optimization steps per field is estimated at 8.96 GPU-hours. Including subsequent student training, the projected cumulative costs across five configurations are 9.15 GPU-hours for TS-TT and 44.28 GPU-hours for E2E-TT, as shown below.

Table 1. Projected cumulative training cost across student configurations. Costs are in GPU-hours on a single NVIDIA H800.

Number of student configurationsTS-TT cost (H800 GPU-hours)E2E-TT cost (H800 GPU-hours)E2E/TS cost ratio
19.008.840.98×
29.0417.711.96×
39.0726.562.93×
49.1235.423.89×
59.1544.284.84×

TS-TT includes the estimated one-time teacher-generation cost of 8.96 GPU-hours and subsequent cached-label training. E2E-TT configuration 1 is measured; configurations 2–5 are projected from configuration-specific timing probes. This fixed-cache cost analysis uses label supervision and is separate from the original main-table recipes.

End-to-end ablation. Following the reviewer’s suggestion, we trained an E2E-TT variant that predicts virtual observations and directly optimizes the translator through the frozen reconstructor using the field loss, without teacher-generated labels:

\widehat X=F_{\mathcal A}(z),\qquad \min_\phi\mathbb E_{\mathrm{train}} \left[\ell_F\left(F_{\mathcal A}(T_\phi(\mathcal B,b;\mathcal A)),X\right)\right],$$ where $F_{\mathcal A}(z)=R_\theta(\mathcal A,z)$ remains frozen and only the translator parameters $\phi$ are updated. **Table 2. Reconstruction errors of the TS-TT and the E2E-TT variant. Lower is better.** | Setting | TS-TT | E2E-TT | | --- | ---: | ---: | | NS–Uniform Transformer | 0.2312 | **0.2208** | | NS–Scan FNO | 0.3449 | **0.3286** | | NS–Scan Mobile-SHRED | 0.3971 | **0.3199** | | Plate Transformer | 0.1538 | **0.0694** | | Pollutant SHRED | 0.2303 | **0.2226** | | Pollutant FNO | **0.2427** | 0.2621 | Each setting retains its original evaluation metric and partition; the TT column uses the original main-table training recipe. Direct field-loss training is effective across these settings. The core contribution of TT is to demonstrate the control capability of the observation input interface for adapting a frozen reconstructor under observation-interface shifts. We now present E2E-TT as another training variant alongside TS-TT, whose two-stage formulation provides explicit interface diagnostics and reusable optimization targets. We have clarified this distinction and include the additional comparisons in the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer s1zQ (3/4)

Comment 2 (W1, Q2): Why use TT rather than fine-tuning or retraining?

The reviewer asks why TT is useful relative to fine-tuning or retraining in the intended deployment setting, and requests a comparison of training time, GPU-hours, and hardware. We address the deployment motivation first, followed by the training-cost evidence.

The key insight of TT is to treat the observation input as a control interface, keeping all parameters of the pretrained reconstructor fixed without requiring additional control interfaces inside the model. This is useful in deployment settings where control over the reconstructor’s internal components is restricted. TT training requires forward evaluations and gradients with respect to its virtual input, ∂ℓ/∂z\partial\ell/\partial z. The optional directional and adjoint derivative interfaces in the FMI standard provide an engineering precedent for exposing input sensitivities through an encapsulated model interface. FMI 3.0.2, Section 2.2.12: Getting Partial Derivatives

For example, Altair’s documentation describes the delivery of proprietary models without making their internal source code available to users. When such models expose the required input-derivative interfaces, these interfaces can support gradient-based optimization of external inputs without opening the internal implementation. This access pattern requires less control over model internals than updating model weights or inserting internal adapters, as in fine-tuning and ControlNet. Altair Twin Activate: Functional Mock-up Interface

Figure 2 illustrates the dependence of reconstruction performance on the training trajectory. By keeping the original reconstructor fixed, TT preserves its original processing path: the translator can be enabled for the new trajectory and bypassed for the original one. Fine-tuning on a new trajectory can also degrade performance on the original trajectory, particularly when the adaptation objective does not preserve performance on the original configuration.

The training costs are reported below, including retraining, LoRA fine-tuning, FiLM, and TT. All listed runs use a single NVIDIA H800 GPU; wall-clock training hours therefore equal GPU-hours. The Plate Transformer results show the accuracy and cost of each reported training recipe:

MethodReported training scheduleTraining cost (H800 GPU-hours)Reconstruction error
LoRA40,000 updates7.510.2361
FiLMReported checkpoint at 20,000 updates3.870.0778
E2E-TT40,000 updates8.840.0694
TS-TTOriginal main-table recipe, including Teacher and DAgger5.670.1538

Its approximately 5.67 GPU-hours include approximately 5.64 hours of Teacher generation and corrective aggregation. The FiLM row reports the same 20,000-update checkpoint used in the additional-baseline comparison.

For the more expensive RU-CFM reconstructor, the recorded costs are:

Method and deployment configurationTeacher + DAgger (h)Student (h)Total training cost (H800 GPU-hours)
RU-CFM retraining, 100,000 updates——67.09
RU-CFM retraining, 100,000 updates——70.50
NS–Scan TT42.830.3243.15
NS–Uniform TT35.510.4936.00

TT takes approximately 36–43 GPU-hours in the listed configurations, compared with 67–70.5 GPU-hours for the listed retraining runs. The table provides cost references for the specified configurations, which use different deployment trajectories. These results illustrate the relevance of input-interface adaptation when the reconstruction core is expensive to train.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers and the Area Chair for their careful assessment. We summarize the main responses and manuscript updates below.

  1. Teacher–student design and E2E training — Comment 1 for Reviewer s1zQ; Comment 1 for Reviewer 2s5N. We clarified TS-TT’s benefits in diagnosing input-interface controllability and reusing teacher supervision, and introduced E2E-TT as another training variant. Section 2.3 and Appendices F.2.3/G.6 now include the formulation, reconstruction comparisons, and costs across multiple student configurations.
  2. Adaptation motivation, baselines, and training costs — Comment 2 for Reviewer s1zQ; Comment 4 for Reviewer CiqA. We clarified how TT differs from retraining, LoRA, ControlNet, and FiLM in its adaptation interface and model-access requirements. The Introduction and Appendix G.7 now include these distinctions and training-cost comparisons. Additional LoRA/FiLM reconstruction results are provided in the rebuttal.
  3. Causal observation access — Comment 2 for Reviewer 2s5N. We added unidirectional-GRU comparisons in four settings. All four causal variants improve over No TT. These results are included in Appendix F.2.4.
  4. Cross-trajectory generalization — Comment 3 for Reviewer s1zQ; Comment 3 for Reviewer 2s5N. We added trajectory-generation rules, parameter ranges, splits, and three-seed results to Appendix F.7, documenting stable gains on both new trajectory instances and a held-out trajectory family.
  5. Training versus deployment requirements — Comment 4 for Reviewer s1zQ; Comment 2 for Reviewer CiqA. We clarified that full-field supervision is used during offline training, while deployment requires only the sensing configuration and its observations. The expanded Figure 1 caption makes this distinction explicit.
  6. Virtual-observation targets and presentation — Comments 1 and 3 for Reviewer CiqA. We explained the target using the original Proposition 1 and A-supervised comparison. We also added explanations of the input Jacobian and DAgger in Sections 2.2–2.3.1 and expanded the Figure 1 caption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.