- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BITEMPO, a test-time latent inference framework for scientific equation discovery, aiming to improve how LLM-based symbolic regression methods leverage historical numerical feedback during iterative search. Specifically, the method constructs a latent search space consisting of global structural preferences and local refinement preferences. By maintaining two probability distributions that are updated at different frequencies, the framework guides structural exploration and local refinement separately. In addition, the authors introduce a lightweight utility surrogate based on ridge regression to select promising latent proposals before invoking the language model.
Critical strengths
- The paper addresses a practically meaningful research problem. Effectively leveraging historical numerical feedback to guide subsequent equation generation remains an important challenge in LLM-based scientific equation discovery.
- BITEMPO adapts its search strategy through test-time updates of latent distributions without modifying the language model parameters. This makes it potentially suitable for integration into existing LLM-based symbolic regression frameworks as a separate search controller.
Critical weaknesses
- The methodological novelty remains somewhat limited. Although BITEMPO employs explicit probability distributions to represent search preferences, the paper does not sufficiently distinguish this approach from existing adaptive search methods. A comparison with a conventional Cross-Entropy Method (CEM) operating in the same search preference space would help clarify the benefits of the proposed framework.
- The effectiveness of the utility surrogate is not sufficiently evaluated. The authors employ a ridge-regression-based utility surrogate to select promising generation preferences. However, the paper does not directly evaluate the predictive accuracy of this surrogate, for example, by reporting the correlation between predicted and observed utilities of latent proposals.
- Performance does not consistently improve with a larger candidate budget. In Table 1, under the Llama-3.1-8B setting, [email protected] on the Material subset decreases from 68.00% with BITEMPO-200 to 64.00% with BITEMPO-400.
- The effectiveness of the two-timescale design is not sufficiently supported by the ablation experiments. Although the paper compares BITEMPO against a Single-Timescale Updates variant, the existing experiments do not conclusively demonstrate the necessity of using different update frequencies for global and local preferences. In particular, the exact update interval used in the Single-Timescale baseline is not clearly specified. Additional experiments with different global/local update intervals, such as (8, 8), (16, 16), (32, 32), and the reversed (8, 32) configuration, would help isolate the effect of timescale separation.
Questions for authors
- Given that the 40-dimensional latent space consists of manually predefined generation preferences, how does BITEMPO fundamentally differ from existing adaptive search methods? What specific advantages does the proposed latent distribution representation provide?
- How accurately does the ridge-regression surrogate predict the actual utility of latent proposals? Have the authors evaluated the correlation between predicted and observed utilities?
- What exact update interval is used for the Single-Timescale baseline? Have the authors evaluated alternative update schedules, such as (8, 8), (16, 16), (32, 32), and the reversed (8, 32) configuration? How can the reported improvement be attributed specifically to timescale separation rather than update frequency?
- In Table 1, why does increasing the candidate budget from 200 to 400 correspond to a reduction in [email protected] on the Material subset? Can the authors analyze whether this result is due to stochastic search behavior or the final candidate selection mechanism?
4 comments
Author Response to Reviewer Hmov (2/3)
Table R2: Actual update counts.
| Condition | U_g | U_l | Update events E | Total block updates U_g+U_l | Coordinate-update volume 30U_g+10U_l |
|---|---|---|---|---|---|
| BiTempo (32,8) | 6 | 24 | 24 | 30 | 420 |
| Count-matched synchronous blocks (13,13) | 15 | 15 | 15 | 30 | 600 |
We added the actual update counts as Table 14 in Appendix H.2 (lines 1317–1332).
The default schedule has the highest OOD accuracy. It updates the local block more often while using the same total number of block updates as the matched control. The matched control has lower mean NMSE. The two metrics therefore favor different schedules.
Surrogate accuracy and component ablations (W2, Q2)
The surrogate predicts Qwen utility better than the median baseline, and removing its ranking in a separate ablation lowers ID and OOD accuracy.
Prediction evaluation. Mean Pearson r is 0.7255, mean Spearman ρ is 0.5871, and MAE is 45.07% lower than the historical-median baseline. The ridge surrogate predicts candidate training utility from preference features computed from each latent. We paired each saved prediction with the utility measured after generation. We restored the prediction scale using only the history available when the prediction was made.
Table R7: Historical online prediction accuracy.
| Model / budget | Prediction pairs | Pearson r | Spearman ρ | Surrogate MAE | Historical-median MAE |
|---|---|---|---|---|---|
| Qwen / 400 | 8,225 | 0.7255 | 0.5871 | 0.5743 | 1.0455 |
We added this prediction evaluation as Table 16 in Appendix H.5 (lines 1379–1391).
Component ablations. The ablations show that removing any one of the three components lowers mean ID accuracy by about 4 to 6 percentage points and mean OOD accuracy by about 7 to 9 percentage points.
We remove each component separately and keep the other generation steps, numerical fitting, update schedules, and final selection rules unchanged:
- Without surrogate ranking: We stop using surrogate scores to rank latents before generation. We keep the proposal-pool size and sampling rules. We described the ranking ablation in Appendix H.4 (lines 1355–1357).
- Without latent-compatible parent retrieval: We stop using latent compatibility to reorder parent candidates and choose parents by quality. We keep the archive rules and the maximum number of parents. We added the parent-selection setting in Appendix H.4 (lines 1357–1359).
- Without residual guidance: We stop using measured residuals to adjust the update targets for local residual coordinates. We keep these coordinates and their elite-based updates, along with the prompt channel, local update schedule, and edit coordinates. We specified the residual-guidance removal in Appendix H.4 (lines 1359–1361).
Table R3: Component ablations (60 ID tasks, 32 OOD tasks). Values are mean ± sample SD across three runs (%).
| Condition | ID [email protected] (%) | OOD [email protected] (%) |
|---|---|---|
| Full BiTempo | 40.56 ± 4.19 | 55.21 ± 3.61 |
| Without surrogate ranking | 36.11 ± 0.96 | 46.88 ± 3.13 |
| Without parent retrieval | 34.44 ± 0.96 | 45.83 ± 1.80 |
| Without residual guidance | 35.56 ± 2.55 | 47.92 ± 1.80 |
Relative to the full method, mean ID accuracy drops by 4.45, 6.12, and 5.00 percentage points without surrogate ranking, parent retrieval, and residual guidance, respectively. Without surrogate ranking, the three runs succeed on 15, 16, and 14 of the 32 OOD tasks. Without parent retrieval, they succeed on 15, 15, and 14. Without residual guidance, they succeed on 15, 15, and 16. Mean OOD accuracy is 8.33, 9.38, and 7.29 percentage points below the full method. We added these ablation results as Table 15 in Appendix H.4 (lines 1363–1378).
Removing pre-generation ranking reduces mean ID Acc by 4.45 percentage points and mean OOD Acc by 8.33 percentage points. Table R7 measures how well the predictions match actual utility. Table R3 measures how removing each component changes search accuracy.
Author Response to Reviewer Hmov (3/3)
Material budget effects and final selection (W3, Q4)
A larger budget improves average accuracy in the repeated runs, but the shortlist can drop successful candidates. Material [email protected] fell from 68.00% to 64.00% in the original table, a loss of 1 successful task. In two supplementary runs under the same inference environment, mean ID Acc rose from 56.00% to 66.00%, and both seeds improved with the larger budget. We added these budget-comparison results in Appendix H.8 (lines 1459–1463).
The shortlist can exclude a successful candidate that is still in the full pool. We reran final selection at the 200/400 cutoffs of the same search trajectories. The 200-slot cutoff here is a prefix of the 400-candidate trajectories, so its accuracy (60.00%) differs from the separate 200-candidate runs above (56.00%). We compared the candidate pool, shortlist, and final output. We used external labels only to evaluate the results after search. We revised lines 292–294 of the manuscript to explain this final-selection analysis.
Table R8: Candidate success coverage and final selection on the same trajectories.
| Evaluation split | Cutoff | Candidate-pool success coverage (%) | Shortlist success coverage (%) | Final Acc (%) |
|---|---|---|---|---|
| ID | 200 slots | 82.00 | 68.00 | 60.00 |
| ID | 400 slots | 82.00 | 66.00 | 66.00 |
| OOD | 200 slots | 92.00 | 84.00 | 76.00 |
| OOD | 400 slots | 94.00 | 82.00 | 82.00 |
Along the same trajectories, ID and OOD each gained 4 successes and lost 1, for a net gain of 3. The full pool kept or increased its coverage of successful candidates, but the shortlist lost coverage. Adding candidates can therefore change which ones pass the final filtering step. We added these results as Table 20 in Appendix H.8 (lines 1464–1480) and included Figure 7(c), whose caption appears in lines 1500–1503.
In both cases that lost success, the earlier successful candidate remained in the pool but was missing from the shortlist. When we added it back, the same training-based selection rule chose it again. The shortlist step caused these losses. A larger budget can therefore improve average accuracy while the final selection loses success on individual tasks. We added this add-back diagnosis in Appendix H.8 (lines 1480–1485).
Thank you and we are looking forward to your feedback!
Dear Reviewer Hmov,
Thank you again for your valuable comments, which have greatly improved our paper! We have carefully addressed your concerns. As the discussion phase nears its end, we would be deeply grateful if you could kindly share your thoughts on our rebuttal and consider raising your score. We truly appreciate your time and effort in reviewing our work.
Best regards, Authors
Author Response to Reviewer Hmov (1/3)
Dear Reviewer Hmov:
Thank you for reviewing our work. We respond to each point below.
Preference space and CEM controls (W1, Q1)
BiTempo lets local preferences respond quickly to feedback while keeping the global search distribution fixed between updates. BiTempo keeps separate distributions for structural and refinement preferences. It updates them at different intervals using the same candidate evaluations. The CEM control updates all coordinates together. BiTempo can update the local distribution while keeping the global distribution fixed. Measured residuals guide the local updates.
We use the same sampled latent to guide generation and choose compatible parents. Previously evaluated candidates provide reference programs. Their preferences and scores also guide future proposals. CEM fits the distribution, and a diagonal Gaussian represents its state. The surrogate ranks latent proposals before generation, and retrieval supplies previously evaluated programs. BiTempo connects these steps through the preference state. Candidate feedback updates its global and local distributions at different rates, and each sampled latent guides both generation and retrieval. We have revised the manuscript to clarify these methodological contributions in lines 186–190 and Appendix H.1 (lines 1245–1259).
We define the meaning of each preference coordinate by hand. The 40-dimensional space gives us explicit controls over generation. Its 30-dimensional global block represents variable, operator, interaction, and complexity preferences. Its 10-dimensional local block represents residual responses and edit preferences. Candidate feedback changes the means and scales of their distributions during search. These changes shift the sampling probabilities, adjust the range of exploration, and control how quickly structural and refinement preferences respond. We added this explanation of the preference coordinates to Appendix H.1 (lines 1259–1274).
BiTempo has higher ID and OOD accuracy than Standard CEM and Joint-CEM in the same preference space. We use the same generation interface, numerical evaluation code, candidate budget, and final selection rules for all methods. Standard CEM fits the distribution to elites from the current generation. Joint-CEM keeps the other mechanisms and updates all coordinates every 8 feedback observations.
Table R9: CEM comparison in the same preference space (60 ID tasks, 32 OOD tasks).
BiTempo has the highest ID/OOD accuracy. Relative to Standard CEM, ID/OOD accuracy increases by 13.33/6.25 percentage points, respectively. Relative to Joint-CEM, the increases are 16.67/6.25 percentage points, respectively. We added this CEM comparison as Table 12 in Appendix H.1 (lines 1279–1295).
Update schedules and counts (W4, Q3)
BiTempo has higher OOD accuracy than the synchronous control when both use the same number of block updates. The global block controls structural preferences. The local block controls residual and edit preferences. The global parameters stay fixed between global updates. Each round can still sample a different latent and choose different parents, so equation structures can change. We revised lines 57–67 of the manuscript to clarify this distinction.
We set K_g=32 and K_l=8 to update local preferences more often than global preferences. This lets local preferences respond faster to residual and edit feedback. Both blocks use the same candidate history. We count each interval in latent feedback observations. Once we have enough samples and valid elites, each global period has four opportunities for a local update. We added this explanation of the update intervals to lines 246–249 of the manuscript.
Table R1 compares the default (32,8), synchronous (16,16), and reversed (8,32) schedules, along with (13,13), which matches the total number of block updates.
Table R1: Effect of update schedules on OOD performance.
We added this schedule comparison as Table 13 in Appendix H.2 (lines 1297–1316).
U_g and U_l count updates to the global and local distributions. E counts events that update at least one block. Updating both blocks at once counts as one event and two block updates.