- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper starts from the observation that a robot can predict future motion accurately while describing its own motion incorrectly. For example, a robot may estimate its speed and direction, then predict future positions using a rule that ignores acceleration. In this situation, a less accurate estimate of physical motion can compensate for limitations in the prediction rule. Improving predictions therefore does not necessarily improve the accuracy of the underlying physical description.
The authors ask when this conflict is unavoidable under a given prediction rule and specified error limits. They require one shared physical estimate to support several predictions. They distinguish three possibilities: an estimate can satisfy all requirements; predictions can succeed only with a physically unacceptable estimate; or no estimate can satisfy all predictions simultaneously. To establish the second case, they find an estimate that satisfies every prediction requirement and use mathematical bounds to rule out every physically acceptable estimate. In such cases, improving motion estimation alone cannot solve the conflict.
Experiments with recorded robot movements (RoboMNIST) show that systems trained to improve future predictions sometimes succeed by producing estimates whose physical errors necessarily exceed the allowed limits (18 of 190 such windows for the affine prediction rule). The conflict persists even with more flexible prediction rules that reduce average prediction error.
The authors then test changes to the model itself. They add acceleration and adjust the prediction rule to account for how motion was measured over a recent time interval. This restores simultaneous accuracy of physical estimates and predictions in many previously impossible cases. They also audit robot calibrations published by other authors (MUKCa, and a CTU dataset with laser-tracker measurements), showing that the calibrated models can predict known distances accurately using unchanged recorded joint angles, where the original models could not succeed within the allowed angle adjustments.
The repairs are imperfect: some previously successful cases become unsuccessful, and the conclusions depend on the chosen error limits and assumptions about measurement uncertainty.
Critical strengths
The paper's main strength is its rigorous distinction between inaccurate physical estimates and conflicts built into the prediction rule itself. It can establish when every estimate that satisfies the prediction requirements must exceed the allowed physical error. This shows precisely when retraining the estimation component alone cannot resolve the problem while the prediction rule and requirements remain fixed. Another strength is evaluating changes on the same individual cases: the authors distinguish whether a satisfactory solution becomes possible from whether the tested system actually produces one, while also reporting previously successful cases that deteriorate. These checks reveal trade-offs that average prediction errors can conceal. Applying the approach to both learned motion prediction and robot calibration gives the diagnosis concrete meaning through explicit physical quantities and error limits. Finally, the certificates are verified carefully (independent high-precision recomputation, pre-registered evaluation protocol for the acceleration study), and the paper is transparent about which analyses were added retrospectively.
Critical weaknesses
-
The paper's main weakness is the limited breadth of its learned prediction experiments. These focus on two test activities, two robots of the same type, short prediction horizons (0.1 and 0.2 s), and relatively small models. Although the calibration studies provide additional evidence, it remains unclear how frequently the diagnosed conflicts occur in more complex robotic tasks. The RoboMNIST experiments also use recorded motion feedback without evaluating estimators based on video, leaving their relevance to systems combining several sensors insufficiently tested.
-
The tolerances are not independently justified, and the conclusions depend strongly on them. The benefits of the proposed acceleration extension depend on the chosen error limits: tightening the acceleration requirement from 0.5 to 0.25 m/s^2 reverses its advantage over the data/budget-matched comparison model in all four settings (Appendix D.1, Table 13), and this matched control was itself added after inspecting the confirmation results. The relative gain over ordinary constant acceleration is also small (1.11 to 2.78 points, and negative at medium speed). The same issue is more pronounced in the calibration audit: a joint tolerance of 0.01 degrees appears very small relative to typical joint-level errors on such arms (encoder offsets, elasticity, backlash), and the certified number of D cases depends heavily on it (for CTU, 99 old D poses at 0.01 degrees but only 21 at 0.05 degrees, with zero recovered; Table 19). With such tight tolerances, certifying "necessary distortion" for an uncalibrated model is close to restating that the model has kinematic errors larger than 0.01 degrees, which is the reason for calibrating it in the first place.
-
The main learned-readout result is fragile under reference uncertainty. Of the 18 certified D windows where the task-trained readout succeeds, only 4 remain certified under an assumed 1 mm position error of the reference trajectory, and none under 1 mm plus a 2 ms timing error (Table 9). The paper notes that no per-point accuracy bound exists for the RoboMNIST references (Appendix A), so it is unclear whether the realized distortion reflects the robot's motion or errors in the reference itself. Similarly, in the repair study all paired certificates vanish at 2 mm (Table 14), and at 1 mm the residual head falls behind the matched control in every setting (Appendix D.2).
-
The calibration audit evaluates models that the authors did not fit, on records whose role in the original model development is unknown (Section 5.1: "The available metadata leave the records' use during other stages of original model development unknown"). The 98 to 100% recovery rates on FR3 and Panda may therefore be partly in-sample. On Kinova, the calibrated model recovers 211 of 1,019 D pairs while losing 107 previously passing pairs, which deserves more discussion than it receives.
-
The paper is very dense and introduces a lot of its own terminology (readout, interface, contract, semantic ball, D-task success, D to J), which makes it hard to follow without repeatedly consulting the appendices. A single worked example carried through the paper (e.g., the constant-velocity example from Appendix B.2) would help considerably.
Questions for authors
The title says "When Correct Predictions Require Incorrect States".
-
What general lessons can a model designer draw from your experiments about when accurate predictions require inaccurate physical estimates, and which model changes can resolve this conflict? Which observed patterns support these lessons, and under what conditions would you expect them to apply to a new task?
-
How were the tolerances (0.1 m/s, 20 mm, 0.01 degrees, 1 mm) chosen? Is there an application or hardware argument for them (e.g., encoder resolution for the 0.01 degree joint tolerance), and how would the main conclusions change under tolerances matched to realistic measurement errors?
-
Given that only 4 of the 18 D successes survive a 1 mm position error assumption, what evidence do you have that the RoboMNIST reference trajectories are accurate at the sub-millimeter level?
-
Were the MUKCa records used in the audit (the 14 non-KPI locations) used at any stage of fitting the published calibrated models? If so, how should the recovery rates be interpreted?