- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes to compress a GP teacher p into a fixed-size surrogate q made of K pseudo-observations with per-point noise. q is fitted by minimizing the grid-averaged pointwise KL(p‖q) under sufficient conditions. The paper’s value proposition is a small, fast GP surrogate that comes with a machine-checked, whole-domain proof that its predictive band sits on a chosen side of the teacher’s.
Critical strengths
-
The question of which side of the teacher’s uncertainty a compressed GP lands on is relevant for safety-critical deployment.
-
The proofs are correct, in my opinion.
Critical weaknesses
-
The manuscript reads very artificially. I recommend rewriting.
-
Theorems 1–2 reduce to the monotonicity of f(r) = ln r + 1/(2r²) − ½ in the ratio r = σ_q/σ_p. Equality holds at the trivial point μ_q = μ_p, σ_q = (1 ± δ)σ_p. “The optimizer matches the bound” therefore only shows σ_q ≈ 1.15σ_p everywhere.
-
The practical scope is weak: It is unclear to me whether this is a practical solution; the speed-up is not practically relevant. In the paper's example (Table 5), the teacher can be trained and queried on-board for predictions.
-
I am unsure how this methodology behaves in high-dimensional spaces. As written, it seems only dimensions 1 and 2 are practical.
-
All tables should report RMSE, NLL, and held-out coverage (in addition to speed-up) across baseline methods for the main experiments.
Questions for authors
What is the highest dimensionality you would consider being practical for the methodology? Why?
2 comments
Response to Reviewer Fseq (2/2)
Weakness 5 (RMSE, NLL and held-out coverage across baselines). We computed these metrics during the response period. In short, the constrained fit and post-hoc scaling keep the teacher's accuracy (RMSE within 2 % on every multi-input problem) and have wider bands. These show as coverage at or above the teacher's and, where the teacher's band was already wide enough, as a higher NLL. Coverage is the share of held-out observations inside a model's own µ ± 2σ band (95.4 % if calibrated). For the synthetic one-input problem of Table 4, with every baseline, we drew 1,000 fresh trajectories for each of 20 seeds, none of them seen by the teacher:
| Method | RMSE | NLL | Held-out coverage of µ ± 2σ |
|---|---|---|---|
| Teacher | 1.1282 | 1.632 | 0.891 |
| Unconstrained surrogate | 1.1281 | 1.632 | 0.891 |
| Post-hoc scaling | 1.1281 | 1.577 | 0.933 |
| Penalty fit | 1.1257 | 1.591 | 0.939 |
| SQP, envelope | 1.1279 | 1.578 | 0.932 |
| SQP, monitor | 1.1280 | 1.777 | 0.830 |
| SQP, faithful | 1.1281 | 1.632 | 0.891 |
RMSE differs by at most 0.0025 between rows. The envelope surrogate covers more held-out points than the teacher and has lower NLL, because the teacher under-covers on this problem. The monitor surrogate narrows the band by design, and its coverage drops.
For the five multi-input problems of Table 2 we scored the teacher, the constrained fit and post-hoc scaling on held-out data (teacher / constrained fit / post-hoc scaling). The surrogates are refits at the settings of Table 2 and carry no certificate. The variance includes the teacher's fitted observation noise for all three models:
| Problem (inputs, held-out points) | RMSE | NLL | Coverage of µ ± 2σ |
|---|---|---|---|
| Meuse 2D (2, 31) | 0.339 / 0.333 / 0.338 | 0.40 / 0.52 / 0.50 | 1.000 / 1.000 / 1.000 |
| Synthetic 3D (3, 4,096) | 0.090 / 0.092 / 0.090 | −1.12 / −0.29 / −0.78 | 0.942 / 1.000 / 1.000 |
| Meuse-3D (3, 31) | 0.320 / 0.326 / 0.318 | 0.26 / 1.23 / 0.94 | 0.935 / 1.000 / 1.000 |
| Ishigami (3, 4,096) | 0.572 / 0.568 / 0.567 | 0.80 / 2.22 / 1.53 | 0.991 / 1.000 / 1.000 |
| Synthetic 4D (4, 4,096) | 0.126 / 0.128 / 0.124 | −0.79 / 1.29 / 0.73 | 0.948 / 1.000 / 1.000 |
RMSE stays within 2 % of the teacher's on every problem. The surrogates' NLL is higher than the teacher's on all five, because their bands are wider than these data need.
On the 42 KIOS and Terra channels we held out one fifth of the trajectories of each regime and rebuilt the teacher from the others. The constrained fit's RMSE is 1.001 times the teacher's in the median (0.889 to 1.462), and post-hoc scaling's 1.000 times (0.963 to 1.040). With the variance as in the paper (no observation noise added), the 2σ band covers on average 98.9 % of the held-out observations for the teacher and 99.9 % for both surrogates. Both surrogates have a higher NLL than the teacher on 40 of the 42 channels.
During the response period, we also scored the penalty fit and both Appendix F SVGP baselines on the same 42 held-out channels, with equal weight per channel:
| Method | Median RMSE / teacher | Median NLL minus teacher (nats/observation) | Mean 2σ coverage |
|---|---|---|---|
| Penalty | 1.005 | 0.163 | 99.9% |
| Post-hoc SVGP | 1.000 | 0.140 | 99.9% |
| Constrained SVGP | 1.000 | 0.141 | 99.9% |
For the five multi-input problems, the penalty scores equal the constrained rows above because those refits used the released penalty recipe. We added a full-covariance SVGP with a separate lengthscale per input, trained for 2,000 Adam steps; this extends Appendix F's one-input baseline and uses the same held-out split and observation noise. The entries below are post-hoc SVGP / constrained SVGP:
| Problem | RMSE | NLL | 2σ coverage |
|---|---|---|---|
| Meuse 2D | 0.335 / 0.373 | 0.48 / 2.42 | 1.000 / 1.000 |
| Synthetic 3D | 0.182 / 0.148 | -0.31 / -0.52 | 1.000 / 0.978 |
| Meuse-3D | 0.339 / 0.448 | 0.96 / 4.47 | 1.000 / 1.000 |
| Ishigami | 0.551 / 0.559 | 1.63 / 2.09 | 1.000 / 1.000 |
| Synthetic 4D | 0.171 / 0.167 | 0.93 / 0.46 | 1.000 / 1.000 |
These added fits have no certificates; iteration-limited or grid-infeasible fits remain included.
We have revised the paper accordingly:
- Table 4 has these columns, and Tables 29 and 30 report the held-out results, including the additional baseline scores.
- The opening sentence of the introduction now reads: "Deploying a Gaussian process (GP) trained on a robot's sensor data requires compressing the full posterior into a fixed-size surrogate, because the time and memory of one GP prediction grow with the number of training observations, and the teacher grows with every trajectory that the robot observes."
- Appendix J states that three inputs are practical for offline updates on the problems of Table 2, that four inputs need an overnight run, and that five or more do not fit an overnight run at these settings.
Response to Reviewer Fseq (1/2)
Weakness 1 (the manuscript reads artificially, rewriting recommended). We agree that the presentation was dense in places, and we have revised it. The abstract is rewritten. The introduction now says that the teacher grows with every trajectory that the robot observes. Contribution 2 and the paragraph after Theorem 2 now say in words when the bounds are attained and how close the fit comes. Contribution 3 states in one sentence what a certificate proves, and the captions of Figure 1 and Tables 1 and 4 now say in words what is shown.
Weakness 2 (Theorems 1 and 2 reduce to the monotonicity of f(r), and matching the bound only shows σq ≈ 1.15σp). We agree that both theorems follow from the Gaussian KL formula and the monotonicity of f(r). Their role in the paper is to give the experiments a reference value: no surrogate with σq ≥ (1 + δ)σp on the grid has KL below Φ(δ), so the excess of a fit's KL over Φ(δ) is the most that any other such surrogate could gain. A KL close to Φ(δ) therefore shows more than σq ≈ 1.15σp: it shows that K points reproduce the teacher's mean and the shape of its variance over the whole domain, and that the optimizer added no width beyond the margin that the contract requires.
How close our optimizer comes depends on K. At K ≥ 25 the constrained fit's KL, averaged over three seeds, is 2.1 to 6.8 % above Φ(0.15) at every teacher lengthscale that we tested. When K is too small for a short lengthscale (K ≤ 10 at lengthscale 0.05), the gap is large. The table of these gaps is in our reply to Reviewer F7tJ (weakness 2), and the revised contribution 2 states where the fit is close to the bound.
Weakness 3 (the speed-up is not practically relevant, and the teacher of Table 5 can run on board). Table 5 shows the teacher at one size, but the teacher does not stay at that size. It keeps every training point, so each trajectory that the robot observes adds to its memory and to its prediction time (caption of Table 5). The surrogate (POGP, K = 25) stays at 79 floats and a fixed prediction time. We measured the growth during the response period: with trajectories of 200 points each, one batch of 200 teacher predictions takes 2.8 ms with 7 trajectories and 3.4 s with 10,000, on one CPU core and with every factorization precomputed, while the fixed-size surrogate takes about 0.08 ms at both of these sizes. For scale, the public GeoLife collection holds 17,621 GPS trajectories with more than 48,000 hours of data. A robot that learns online leaves the size of Table 5 behind, and the fixed-size surrogate is what enables on-board prediction as the data grows. The introduction now opens with this motivation (item 2 at the end of our second comment).
Weakness 4 and the question (high dimensions, highest practical dimension). The paper's target is a robot model indexed by time: one input, with one or six outputs. During the response period, three selected scalar refits certified in 8.3 to 135.8 s with 16 processes on a 20-core desktop processor, permitting in-mission recertification of these channels (our reply to moUG gives the joint six-output timings separately). For more inputs, the highest practical number with the current certifier is three for offline updates and four for an overnight run. Two inputs took under 3 min on 8 workers, three took 16 min to 10.6 h, and four took 13.4 h (Table 2). For five inputs we have no measurement. A straight line fitted to log10 worker-seconds against the number of inputs (the five runs with two to four inputs) gives about 18-fold growth per added input. Extrapolated with ideal scaling over workers, five inputs would need about 156 h on 32 workers. A certificate within one hour would need about 15 workers at three inputs, 270 at four and 5,000 at five; within ten minutes, about 90, 1,650 and 30,000 (the fit's spread is a factor of 6.3 either way). The limit lies in the certificate: it tiles the input domain with boxes that are checked independently, so more workers shorten it (table in our reply to Reviewer moUG, weakness 1). The KL bounds and the constrained fit hold for any number of inputs.