- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SciDialect proposes symbolic compression as an intrinsic, label-free reward for scientific discovery agents. A proposer rewrites the current "scientific state" as compact symbolic expressions over a primitive dictionary. Candidates pass audit gates (independent reconstruction, anchor grounding, cross-decoder consistency) and are ranked by description-length reduction, grounding score and consistency. The selected state then conditions the next solver call.
Critical strengths
- Consistent PreScience F1 gains with paired intervals. Token-F1 improves by 2.3–3.9 points over paired backbone-specific comparators, with paired 95% CIs excluding zero ([2.0, 2.7], [3.4, 4.1], [3.5, 4.2], ).
Critical weaknesses
- Theory doesn't cover the implementation, by the authors' own admission: tokenizer length isn't a proper codelength, decoders don't give independent samples, and the conclusion concedes the assumptions are "stronger than the numerical gates of the actual implementation."
- DiscoveryBench (+1.9 over Direct) is not significant the headlined +4.1 is against the weakest baseline. Single-call setup means no ranking among proposals and no consistency term — the claimed mechanism is partly switched off.
Questions for authors
- Which theorem applies to the implemented selection rule, given the concessions on theory vs implementation?
- Paired test vs Direct with an independent evaluator? Qwen3.5 or Qwen3.6?
- Why a different comparator per backbone? Where is the four-call GLM config defined?
- How do the backbone-free optimization gates work without an LLM, and why no CIs in Table 4 despite 20 seeds?
2 comments
Rebuttal response (2/2)
Q3. Why different comparators, and where is the four-call GLM configuration?
Thank you for asking us to make the comparison explicit. Table 1 evaluates the same ten-baseline pool on all three backbones. Qwen CoT and DeepSeek Retrieval Only are their families' strongest Table 1 baselines. The additional GLM comparison uses three independent candidate-generation calls followed by one reviewer-selection call, all with GLM-4-32B-0414. It is separate from the ordinary Reviewer Rerank row in Table 1; the revision adds this schedule to Appendix H.
The submitted paired gains are +2.3 [2.0, 2.7], +3.8 [3.4, 4.1], and +3.9 [3.5, 4.2], respectively. Appendix H reports the four-call comparator's 15.7 F1, 21.7 grounding, 3,977 tokens, and 0.0131 cache-free cost, versus SciDialect's 19.6 F1, 38.5 grounding, and 0.0029 cost. SciDialect therefore improves F1 and grounding while using approximately 22% of this stronger GLM comparator's cost.
Q4. How do numerical gates work without an LLM, and where are the twenty-seed intervals?
We appreciate the request for concrete implementation and uncertainty details. Structured numerical states permit programmatic checks. The supplied code checks coordinate bounds in [−1, 1] and, for SPO/TMC, an absolute coordinate mean of at most 0.65. SciDialect clips proposals and recenters SPO/TMC vectors, applies a 100-point infeasibility penalty to the minimized acquisition, and permits anomaly-driven principle additions only for valid states with configured anchor/reconstruction proxies of at least 0.70. These are numerical configuration checks; the base optimizer retains its ordinary objective evaluations. Table 4 uses the same deterministic surrogate, 24 objective evaluations, and eight proposals per selection step across methods.
The paired 20-seed uncertainty for the Table 4 configuration is:
| Task | Endpoint gain over PiEvo-core [95% CI] | Trajectory-area gain [95% CI] |
|---|---|---|
| MBO | +4.2 [1.2, 7.2] | +10.4 [4.1, 16.7] |
| NHO | +2.8 [0.4, 5.2] | +3.3 [0.9, 5.7] |
| SPO | +9.7 [5.1, 14.3] | +14.4 [7.3, 21.5] |
| TMC | +10.3 [4.6, 16.0] | +4.5 [1.4, 7.6] |
All intervals are positive, while the published Table 4 means remain unchanged. SciDialect also has 100% valid proposals and zero violations on all four tasks; the corresponding PiEvo-core validity rates are 55.8%, 81.2%, 66.7%, and 49.6%. These results provide both uncertainty and a concrete implementation account for the numerical experiment.
Thank you again for helping us make the implementation and its evidence more precise. We sincerely expect your reassessment based on these improved evidences.
Rebuttal response (1/2)
Thank you for the precise questions and for recognizing the consistent PreScience gains. Your feedback helps us make the implementation-level evidence clearer. We address the theoretical correspondence, the full selection mechanism, and the requested statistical and protocol details below.
W1 / Q1. Which result applies to the implemented selection rule?
Proposition 2 in Appendix D.1 directly covers the implemented compression-and-dictionary component. For a fixed proposition batch,
=\frac{w_c}{N(P)}r_{\mathrm{dial}}(d_0\to d_c;P,K).$$ Here, $N(P)$ is the positive batch normalizer, so the candidate-independent factor preserves the ordering of this objective. The identity applies to the implementation's length measure. Theorem 1 gives its predictive interpretation under the stated coding and verification assumptions; Theorem 2 establishes decision-information preservation under exact recovery of the declared state. The practical reward additionally includes reconstruction, grounding, and agreement. **The fixed-candidate ablation below tests the effect of compression within that implemented reward**, with those terms and all admissibility checks held fixed. This connects the formal compression objective to a direct test of its downstream contribution. ### W2. What does the original DiscoveryBench comparison establish? Table 2 reports the highest mean for SciDialect: **15.8 versus 13.9 for Direct**, a **1.9-point absolute or 13.7% relative gain**. The **+4.1** comparison with Prompt Compression addresses the targeted question of whether shortening context alone is sufficient. The new full-controller study in Q2 supplies the requested paired comparison with Direct and stronger refinement baselines under an evaluator outside the generator family. ### W2. Does the one-call adapter test ranking and consistency? Your distinction between admission and ranking is helpful. The compact one-call adapter tests admission; the full-controller comparison tests candidate selection. Section 3.5 and Appendix A.9 describe the compact realization. In the new GLM-5.3 comparison, we fix candidate proposals, decoder views, admissibility gates, evidence, and the solver, and vary the selection rule: | Selection / solver presentation | Final score | | --- | ---: | | Uniform among admissible candidates / expression + decoding | 34.0 | | Zero compression weight / expression + decoding | 36.0 | | Full intrinsic selection / expression + decoding | **39.0** | | Zero compression weight / decoded text | 34.8 | | Full intrinsic selection / decoded text | **37.5** | The matched contrasts are **+3.0 points** with identical combined presentation and **+2.7** with decoded presentation. This design tests the contribution of selection while holding admissibility fixed, including when both selected memories are presented in ordinary language. Zero agreement weight scores **38.6**, and removing the agreement gate scores **38.2** under the same frozen setup. Selection receives no benchmark answers, evaluator scores, or human feedback. ### Q2. Paired comparison with an independent evaluator; Qwen3.5 or Qwen3.6? Thank you for identifying the inconsistent model label. The original Table 2 uses Qwen3.6-35B-A3B for generation and Qwen3.5-35B-A3B for evaluation. The revised protocol sentence states these roles explicitly. The following is a separate full-controller study with GLM-5.3 generation on all **191 held-out tasks** and a frozen **DeepSeek-V4.1-Flash evaluator**. It compares SciDialect with Direct and expenditure-matched Direct-plus-refinement and refined-RAG controls. The rubric and analysis are fixed before grading; paired intervals resample tasks while retaining all repeats within each task. | New GLM-5.3 protocol | Independent evaluator score | Full minus comparator [95% CI] | | --- | ---: | --- | | Direct, one pass | 29.0 | **+10.0 [6.3, 13.7]** | | Direct + self-refinement | 31.0 | **+8.0 [4.2, 11.8]** | | Refined RAG | 32.0 | **+7.0 [3.5, 10.5]** | | SciDialect full controller | **39.0** | — | The primary independent evaluator gives positive paired gains against all three controls. In particular, the **+7.0 [3.5, 10.5]** gain over expenditure-matched refined RAG directly tests whether the benefit survives a stronger semantic-memory comparison. These paired estimates concern the new full-controller protocol. We have added these paired independent-evaluator comparisons to **Appendix L.1**, alongside the fixed-candidate selection/rendering controls and the refinement baselines’ complete online expenditure. The revised manuscript thus directly documents both the contribution of intrinsic selection and the resulting gains under independent evaluation.