- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper distinguishes between "posterior mean tracking" and "adaptive adjustment for observation reliability." The main diagnostic method is to apply a symmetrical perturbation to the current reading while keeping the displayed historical data unchanged, and then compare the finite difference response with the exact Kalman gain under different observation variance settings. Through three checkpoints, approximately 5%–8% of the required endpoint gain change under the baseline decoding condition can be achieved. Through three specific checkpoints, it is possible to achieve approximately 5%–8% of the required adjustment in terminal gain under baseline decoding conditions. A "held-out reasoning protocol" is shown to improve both local responses and the posterior mean. Furthermore, this paper evaluates a system that extracts linguistic reports and applies an explicit Kalman filter, providing a comparative analysis based on the same stream of extracted data.
Critical strengths
-
A well-defined behavioral diagnostic analysis (see Equations (1)–(2), Sections 2–3, and Appendix B.1) avoids the "shared previous-report error" that can arise in aggregate "effective gain" regression analyses by fixing historical values and employing context-specific exact gains.
-
The study conducted "recovery" tests using the same output results and held-out trajectories.
Critical weaknesses
- The contribution is substantially narrower than the title and system framing suggest. The strongest paired result covers three relatively small checkpoints from two families on scalar linear-Gaussian filtering; successful reasoning recovery is demonstrated on one checkpoint. Existing work already studies Bayesian belief trajectories and discounted filtering. PDDA itself is a language extraction interface followed by a classical estimator, and it exactly matches the generated-code Kalman baseline. or a general limitation of language models. Evidence across more model families and nonlinear settings would materially strengthen significance.
- Some secondary evidence does not identify the mechanism or unconditional capability being discussed. The budget sweep omits complete-pair denominators and treatment of failed pairs. A successful long-budget reasoning protocol therefore does not establish performance at a fixed computational budget. Moreover, schedule GAR divides innovation-energy-weighted fitted gains by arithmetic phase-mean reference gains ; an exact time-varying filter need not self-score to one. These are acknowledged limitations, but the secondary plots must not be read as calibrated mechanistic evidence. Steering readout shifts and oracle-state assistance likewise cannot establish where the original computation fails.
Questions for authors
-
Could you report the full pairwise validity and unconditional error for each budget, as well as a factor-based comparison—distinguishing between "thinking" modes, temperature, and token budgets—within the held-out scenarios?
-
Can the reliability-response gap be replicated using the same pairwise estimator and comparable prompts across other independently developed model families or in non-linear filtering tasks?
-
Regarding the comparison of switching noise, what are the GAR (Generalized Accuracy/Agreement Rate) values obtained using an exact filter, given the same regression model and denominator? Would using a denominator based on "innovation energy matching" alter the conclusions?
-
Can the existing data reproduce all the primary pairwise statistics found in the original invocation dataset? Additionally, could you correct the author list for the reference to Liu et al. provided below?
6 comments
Response to Reviewer xRpV (2/4)
W1-2. Reasoning recovery shown on one checkpoint
Reasoning sharply reduces the weight error, and this now holds for five models from three families. We repeated the held-out experiment of Table 5 on four more models and compared the weight error (local-gain MAE, the mean absolute difference between the model's weight and the optimal weight) under the default and the reasoning settings. Qwen3-32B and Qwen3-30B-A3B enter the reasoning setting by enabling thinking, as Qwen3-8B does. Llama-3.1-8B-Instruct and OLMo-2-7B-Instruct have no thinking mode, so one sentence appended to the otherwise unchanged prompt asks them to write out the calculation step by step before the answer, with a token cap large enough for the full calculation.
Table R3. Weight error of five models under the default and the reasoning settings (Table 30, App. N.3 of the revised PDF).
| Model | Default | Reasoning |
|---|---|---|
| Qwen3-8B | 0.301 | 0.019 |
| Qwen3-32B | 0.179 | 0.017 |
| Qwen3-30B-A3B | 0.225 | 0.082 |
| Llama-3.1-8B-Instruct | 0.247 | 0.038 |
| OLMo-2-7B-Instruct | 0.186 | 0.044 |
Under the reasoning setting, the weight error drops by more than 60% in all five models and falls below 0.05 in four of them, so the recovery under reasoning is not limited to Qwen3-8B. Qwen3-30B-A3B, a mixture-of-experts model that activates about 3B parameters per token, improves from 0.225 to 0.082.
W1-3. Difference from prior work on belief trajectories and discounted filtering
Prior work compares how close a model's whole trajectory of estimates is to the Bayes-optimal trajectory, whereas our audit measures whether the weight that a model gives to a new reading changes with the reliability of the sensor. The latter cannot be seen from trajectory agreement. The reason is that a model that gives every new reading roughly the same weight can still stay close to the optimal estimates on a trajectory with a constant noise level (Section 1). This is what we observe. The estimates of all five models correlate with the exact posterior mean at 0.927 or above (Table 1A), yet when the declared noise variance increases from 1 to 16, the optimal weight should decrease from 0.618 to 0.221, and the weights of the three checkpoints in Table 1B decrease by only 5% to 8% of this amount. In other words, the trajectory agrees well while the weight barely adapts to reliability. The fixed-history audit is designed to measure this weight directly. It shows the model exactly the same history of readings, changes only the noise variance declared in the prompt, and shifts the current reading up and down by δ to see how much the estimate follows. This gives the weight of that single update, which is compared with the exact Kalman gain under the same conditions. As you noted among the strengths, this design also avoids the shared previous-report error of aggregate regressions.
Studies of discounted filtering fit a model's trajectory of estimates with a fixed discount factor. A fixed discount does correspond to a weight that does not change with the noise, but it cannot explain the uncertainty that the model reports. As the noise variance increases from 1 to 16, a fixed discount predicts that the interval widens by a factor of 4 and the exact filter by 2.39, while the model's interval widens by only 1.79 (Section 3, App. B.6). A fixed discount therefore describes the trajectory of estimates but not the complete update of the model. We discuss these studies in Section 7.
W1-4. PDDA and the generated-code Kalman baseline
PDDA and the generated-code Kalman baseline have the same accuracy, which is expected because both carry out the numerical update with the Kalman formula. The difference between them is that PDDA needs no generation for an update. The four updaters compared in Table 2B receive exactly the same readings, which the language model has extracted from the reports, and they differ only in who carries out the update from these readings. When the language model performs the update itself, the estimation error is 0.435, and it is 0.038 when the model reasons first. Whenever the update uses the Kalman formula, either executed by model-generated code or by PDDA, the error is 0.028. This shows that the error arises when the language model performs the numerical update itself, which agrees with the conclusion of the fixed-history audit. Between the two options that use the Kalman formula, the code tool still has to generate about 180 tokens of code per update and takes about 0.45 s, whereas PDDA generates nothing and takes less than 1 ms (Table 6). The role of PDDA in the paper is therefore to show that this finding translates directly into a system design, in which a filter performs the numerical update and the language model only reads the values from the reports.
Response to Reviewer xRpV (3/4)
W2-1 & Q1-1. Budget sweep: complete pairs and the unconditional error
Following your comment, we have replaced the original budget sweep with one that reports every denominator. The results show that reasoning helps only when the token budget is large enough for it to finish.
As you pointed out, the budget sweep in App. B.5 was run on the development set, computed errors over completed pairs only, and did not record how many pairs completed, so its values cannot represent unconditional performance. We therefore re-ran the sweep on the 12 held-out trajectories of Table 5 (Qwen3-8B, thinking enabled, temperature 0.6) and used it to replace the original values in Section 5 and App. B.5. Each weight is computed from a pair of calls, in which the current reading is shifted up and down by δ, and a pair is complete only when both calls return a parsable answer. Each budget has 864 calls and 432 pairs. Pairs that fail to complete are not discarded. They are scored as if the model had not updated its estimate, that is, with a weight of zero. The last column of Table R4 is therefore the unconditional error requested in Q1.
Table R4. Budget sweep on the held-out trajectories (Table 32, App. N.5 of the revised PDF).
| Token cap | Parsed calls | Complete pairs | Weight error, complete pairs | Weight error, all pairs |
|---|---|---|---|---|
| 2,048 | 0% | 0 of 432 | — | 0.410 |
| 4,096 | 1% | 0 of 432 | — | 0.410 |
| 8,192 | 56% | 147 of 432 | 0.014 | 0.282 |
| 16,384 | 100% | 432 of 432 | 0.011 | 0.011 |
| 32,768 | 100% | 432 of 432 | 0.019 | 0.019 |
The reasoning trace of the model has 8,066 tokens on average (App. B.3), so no pair completes with a cap of 4,096 tokens or fewer, and the unconditional error is 0.410. With 16,384 tokens or more, all pairs complete and the error is between 0.011 and 0.019, so reasoning helps only when it can finish. The last row is the protocol of Table 5.
Success at a long budget does not establish performance at a small fixed budget, and Table R4 shows that reasoning does not help when the budget is small. This is the motivation for PDDA, whose update does not generate any tokens.
One additional note on scoring. Truncated calls are counted as unparsed in Table R4. If they are instead parsed with the parser of App. M, only the 8,192 row changes, from 147 to 151 complete pairs.
Q1-2. Separating thinking mode, temperature, and token budget
Among the three factors, temperature has little effect, and the weight error drops only when thinking is enabled with a sufficient token budget. We changed one factor at a time on the held-out trajectories of Table 5, and the results are given in Table R5. The scoring is the same as in Table R4, so pairs that fail to complete are included in the error.
Table R5. One factor at a time on the held-out trajectories (Table 31, App. N.4 of the revised PDF).
| Thinking | Temperature | Token cap | Complete pairs | Weight error, all pairs |
|---|---|---|---|---|
| off | 0 | 512 | 432 of 432 | 0.301 |
| off | 0.6 | 512 | 432 of 432 | 0.287 |
| on | 0 | 32,768 | 427 of 432 | 0.024 |
| on | 0.6 | 2,048 | 0 of 432 | 0.410 |
| on | 0.6 | 32,768 | 432 of 432 | 0.019 |
Without thinking, increasing the temperature from 0 to 0.6 changes the error only from 0.301 to 0.287, so temperature has little effect. In this case all calls finish within 512 tokens, so the cap is not binding and larger caps were not tested. With thinking and 32,768 tokens, the error is about 0.02 at both temperatures. With thinking but only 2,048 tokens, the reasoning cannot finish and the error is 0.410, so the improvement requires thinking with a sufficient budget, and our statements about reasoning in the paper refer to this complete setting. The first and last rows are the two protocols of Table 5.
Response to Reviewer xRpV (1/4)
We sincerely thank the reviewer for the careful and detailed comments, and we appreciate your recognition of the fixed-history design and of the recovery tests on held-out trajectories. For easier reference, we label the points in W1 as W1-1 to W1-4 and those in W2 as W2-1 to W2-3, and answer Q1 to Q4 together with the related points. We have uploaded a revised PDF in which all changes are marked in blue.
The models barely adjust their weight on a new reading to its reliability, and this gap is replicated on other model families and on a nonlinear filtering task.
Two quantities are used below. The weight (Kalman gain) is the share that a model gives to the new reading when it updates its estimate, and the optimal weight decreases as the sensor noise variance R increases. GAR is the decrease in the model's weight as R goes from 1 to 16, divided by the decrease in the optimal weight. It is 1 for a model that adjusts optimally and 0 for a model that does not adjust at all.
Following your suggestion in Q2, we evaluated five additional models with the same paired estimator, prompts, and settings as in Table 1B (δ = 3, 120 paired contexts per declared variance, greedy non-thinking decoding). Together with the three original checkpoints, the audit now covers eight models from four families (Qwen, Mistral, Llama, and OLMo), ranging from 1.7B to 32B parameters.
Table R1. Weights and GAR of eight models at three noise variances (Table 28, App. N.1 of the revised PDF).
As R increases from 1 to 16, the optimal weight decreases from 0.618 to 0.221, whereas the weights of all eight models remain almost unchanged. All GARs are below 0.08, and even the largest upper end of the 95% CIs is only 0.139, so the gap is replicated in all four model families. Larger models do not reduce the gap either, since the GARs of Qwen3-32B (0.040) and Qwen3-30B-A3B (0.033) are no higher than that of Qwen3-8B (0.054). To make sure that the new runs are directly comparable with the original ones, we also re-ran Qwen3-8B and obtained 0.057 [0.012, 0.105], in agreement with the 0.054 of Table 1B.
For the nonlinear task, we replaced the linear dynamics with xt=xt−1+2sin(xt−1/2)+wt and kept all other settings unchanged. The optimal weights are computed by a grid filter with 2,001 points. Table R2 reports four models from three families on this task.
Table R2. Weights and GAR of four models in the nonlinear task (Table 29, App. N.2 of the revised PDF).
In this task the optimal weight decreases from 0.515 to 0.072, while the weight of each of the four models changes by less than 0.02 and the largest GAR is only 0.041. The gap is therefore not specific to linear-Gaussian filtering, which is also in line with the discrete HMM result in Table 20 of the paper.
Taken together, both statements in the title now rest on broader evidence. "Track state" means that the estimates follow the state, and the estimates of the five surveyed models correlate with the exact posterior mean at 0.927 to 0.977 (Table 1A). "Underuse observation reliability" means that the weight is not adjusted to the reliability, which now holds for eight models from four families and for the nonlinear task. We have added these results to Section 3 and to the new Apps. N.1 and N.2 of the revised PDF.