acceptodds
Under review as a conference paper at ICLR 2027

What Survives Learned Symbolic Compression?

Abstract

Lossy text compression for language-model pipelines is judged by reconstruction distance or by downstream task accuracy, and neither says whether the compressed representation still states what the source stated. We measure agreement with a constructed reference for grammar-constrained symbolic codes. The reference represents source facts as equations, and a fixed decoder recovers the equations a code encodes. An exact rule system compares their consequences independently of the code's well-formedness checker. On controlled arithmetic micro-worlds, we evaluate learned translators from 70M to 2.8B parameters against structured controls and learned baselines at matched byte and token budgets. Passing the checker is not preserving the source: a self-consistent mistranslation passes it, and so do sampled learned-translation errors. Across byte budgets, the best learned trace translator trails the strongest compact structured control, with area under the closure-agreement curve of 0.67 against 0.86. Increasing translator size does not close this aggregate gap; the largest Pythia translator matches the best smaller translator's saved byte-axis agreement scores. Source fidelity relative to the constructed reference, internal validity, and what a consumer model recovers are three different quantities and should be measured separately.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
lossy text compression, prompt compression, semantic fidelity, rate–distortion, deductive closure, evaluation protocol, grammar-constrained decoding, context compression, LLMLingua

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DSXgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what learned text compression preserves in exactly solvable affine arithmetic worlds. A construction record projects each source into canonical rational equations, and a fixed decoder maps compressed code into the same language. The evaluation signature contains explicit equations, uniquely determined variable values, and pairwise differences; closure agreement is the Jaccard overlap between source and code signatures. Grammar and internal-constraint acceptance, source agreement, and another model’s value recovery are measured separately. Over-budget outputs remain in the denominator and receive zero budgeted agreement.

Training uses 24,000 worlds, with 800 renderings of 200 held-out worlds and a consumer subset of 240 renderings from 60 worlds. Renderings of the same world remain in one split. LoRA trace translators span Pythia 70M to 2.8B and two Qwen models with three seeds. JSON, CCL, and template-prose controls also have trained encoders; a separate oracle reads the construction directly. Byte and reference-token budgets generate separate outputs at four allowances. CCL-Min achieves byte AUC 0.860 versus the best trace’s 0.672; on the token axis JSON reaches 0.514 and CCL-Min 0.426. Scale findings concern the shared training procedure, while consumer findings depend on reader and decoding conditions. Additional analyses cover redundant renderings, sampled failures, parsing, and stop reasons.

Critical strengths
  1. The source equations and fixed decoder make omissions, changed facts, and derivable redundancy inspectable. In Table 1, the 1B, 2.8B, and Qwen 0.6B failure samples have no budget violations but contain checker-accepted outputs, directly establishing learned source errors that internal checks miss. World-level split isolation and inclusion of over-budget outputs also make the controlled comparison easier to interpret.

  2. JSON and CCL controls use trained encoders, while the oracle separately represents direct access to correct facts. Byte and token conditions generate their own budgeted outputs, and Table 2 shows that the leading format changes with the cost unit. Multiple seeds, scales, and a second model family add useful checks. The paper also distinguishes sampled failures from prevalence estimates and an AUC plateau from model equivalence, while keeping consumer conclusions tied to the stated response conditions.

Critical weaknesses
  1. The signature on pages 2 and 3 includes explicit equations, so a fully equivalent rewriting can lose agreement. Replacing x=2, y=x+3, z=2y with x=2, y=5, z=10 preserves all affine information but loses the explicit member 2y-z=0 under the chosen rules, scoring 6/7. This representation dependence is disclosed, but the main CCL versus trace gap is not checked under equivalent bases, value-and-difference-only scoring, or alternative consequence weights. The 0.860 versus 0.672 result therefore first establishes an advantage for this signature; its interpretation as general source-information preservation needs further validation.

  2. Appendix A.7 on page 16 explicitly states that the current record omits the adapted CCL serialization, decoder, target builder, construction implementations, and raw learned outputs. The descriptions do not suffice to independently reconstruct the key formats. Format overhead, translation errors, omissions, and budget violations all lower AUC. Appendix F’s error labels partly derive from already budget-penalized scores, so they do not adequately separate these causes. This limits both the explanation of which design improves performance and the inspectability of the central measurement study.

  3. The consumer comparison on page 8 pools two budget axes and a smaller world subset, whereas the main closure comparison uses the full report set’s byte axis. Their orderings therefore do not directly show preservation versus readability for the same codes. The paper acknowledges this and already reports parsing, stop reasons, and a lower-reasoning GPT-OSS condition. However, substantial truncation remains under the 256-token cap. Missing information and consumer decoding failures have not been separated on matched codes and worlds, limiting guidance for practical compression choices.

Questions for authors
  1. How do the CCL versus trace ranking and gap change under value-and-difference-only scoring, equivalent equation bases, and alternative consequence weights? Can the contribution of explicit-equation members to the AUC gap be quantified?

  2. Please specify CCL syntax, decoding, and target construction sufficiently for independent implementation, and use raw outputs to distinguish representation cost, translation errors, omissions, and budget violations. Can output-to-checkpoint mapping also be verified?

  3. On the same codes and worlds, which format differences persist when source agreement and consumer recovery are reported together and GPT-OSS has enough output space to finish? Can the existing parsing and stop statistics stratify that comparison?

6 comments

Thanks for the for the detailed analysis and concrete checks.

Signature Decomposition and Worked Example Across Five Scoring Rules (Sections 2 and 5, Appendix B) On the byte axis, CCL-Min leads the best trace translator (TE, 2.8B = TE, 1B) under all five scoring rules—by +0.189 under the paper's signature (a), +0.193 under consequences only (b), +0.085 under explicit equations only (c), +0.360 under exact affine equivalence (d), and +0.124 under values only (e) (gaps are computed from unrounded AUCs)—and for every encoder seed (0.857–0.861 under (a)). The explicit members' contribution (a) − (b) to the CCL-Min − TE, 1B gap is −0.0045 on bytes and −0.0070 on tokens: they do not create the gap. The alternative consequence weightings evaluated are (b), (d), and (e); no other weighting was run.

We ran your value-basis rewriting (x=2, y=5, z=10 against x=2, y=x+3, z=2*y) and the Section 2 fixtures through the exact rational oracle under five rules: (a) paper signature, (b) consequences only (values and pairwise differences; first-run predicates missed non-unit values or differences in 186 decoded records, moving no AUC by more than 4.8e-06), (c) explicit equations only (Jaccard, computed whether or not the decoded system is consistent), (d) exact affine equivalence (1 or 0 per record via RREF row-space equality), and (e) values only: row 142 - 149 in the revised version

Five-Rule Closure-AUC Across All 236,800 Decoded Records (Appendix B)

Token-axis ranking and encoder-seed compliance: On tokens, Canonical JSON leads all learned systems under (a), (b), (c), and (e), while Oracle core leads under (d) (0.202; CCL-Min 0.196, Canonical JSON 0.190). Between CCL-Min and TE, 2.8B, CCL-Min leads under (a) (+0.057), (b) (+0.064), and (d) (+0.196), whereas four traces (TE, 2.8B; TE, 1B; TE, Qwen3 0.6B; TE, Qwen3 1.7B) lead CCL-Min under (c) (−0.020) and (e) (−0.033). Per-seed results (Appendix B) show that CCL-Min seed A stays within token budgets on 769 / 770 / 724 / 800 of 800 records (AUC 0.561 under (a), 0.682 under (e)), whereas seeds B (0.364 / 0.412) and C (0.352 / 0.400) exceed the token budget on 55.8 %–74.1 % of records at 0.30–0.60. Correction in submitted Table 5 and regression check: We correct one scoring error in submitted Table 5's Oracle core byte row from 0.000 / 0.725 / 1.000 / 1.000 (AUC 0.768) to 0.000 / 1.000 / 1.000 / 1.000 (AUC 0.850), caused by the submitted scorer applying the byte penalty to a JSON decoding wrapper on 220 of 800 compliant records at byte budget 0.45; the token row (0.202) and all translator and structured-control rows are unchanged. Separately, re-scoring all 236,800 decoded records matches the recorded per-record unpenalized scores, statuses, and budget flags on 0 records with a difference, reproducing 25 of 26 series × axis AUCs (largest difference 3.7e-15) and 33 of 34 rows of submitted Tables 5, 6, and 8.

For Weakness 2 and Question 2: specifying CCL syntax, decoding, and target construction, verifying output-to-checkpoint mapping, and using raw outputs to separate representation cost, translation errors, omissions, and budget violations.

Specifications, Implementations, and Output-to-Checkpoint Mapping (Appendices A.7, D, K, Supplementary Material)

Syntax, decoders, and target construction (Appendix K, Code format specifications, decoders, and target construction, and supplementary docs/CODE_FORMATS.md): CCL-Min serializes canonical equations on one line as @CCL/1m E=<eq>|<eq>|...; CCL-Core uses header @CCL/1, TYPE=affine-rational-v1, and EQ=<eq> lines. Target construction orders a world's canonical equations and takes the largest prefix fitting the byte or token budget. The supplementary archive provides executable decoders, target and world construction, the rational closure oracle, the three trace checkers, the scorer revision patch, and all 268,800 report records (and strong-encoder records) with source texts, reference equations, raw outputs, 236,800 decoded equation sets, and scorecards. Output-to-checkpoint mapping (Appendices A.7 and D, supplementary results/): Every one of the 249,600 learned records maps with 0 checkpoint mismatches and 0 system-id mismatches to one of the 39 trained cells, which have 39 / 39 / 39 of 39 distinct cell, adapter, and adapter-tarball SHA-256 hashes. The adapter weights are not included; their 39-cell hash table is included. This checks every record's labels against the cell table, and the stored objects' size and checksum against the manifest; for Pythia 1B seed A (cell 7287B0B85BDC, checkpoint-376) and Pythia 2.8B seed A (cell 1D312290FEBA, checkpoint-376), each system's complete report grid was regenerated greedily from the base model and the cell's adapter weights, with the same generation code and batching as the report run, and 12,800 of 12,800 regenerated outputs are byte-identical to the recorded raw outputs (800 of 800 at byte budget 0.45 for each system; 6,400 of 6,400 per system over 8 budget cells), with 0 mismatches.

Reviewer_z6aPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether learned symbolic compression preserves specified source content, separately from whether the resulting code passes an internal checker or enables a consumer model to recover answers. It constructs arithmetic micro-worlds, represents their source facts as canonical affine equations, and evaluates decoded compressed representations using a finite signature of explicit equations, entailed values, and pairwise differences.The experiments compare Pythia and Qwen trace translators with learned structured controls, an oracle, and task-oriented compression systems. The principal findings are that internally valid outputs can disagree with their sources, CCL-Min achieves higher byte-budget agreement than the evaluated trace translators, and canonical JSON leads the structured controls under token accounting. The paper also reports a byte-axis plateau among larger Pythia translators and consumer specific recovery results.

Critical strengths
  1. The study makes source preservation independently testable in a controlled setting. The source reference is constructed before compression and is compared with decoded equations using exact arithmetic. This separates the question of whether a code describes the intended source from whether it is internally acceptable.
  2. The budget comparisons provide a useful result with reasonably matched learned controls. The study evaluates both byte and token allowances, retains over-budget outputs in the denominator, and keeps renderings of the same latent world together across data splits. The structured controls use trained Pythia 1.4B encoders; their advantage is therefore not simply a comparison between learned translators and an oracle that receives the correct construction directly.
Critical weaknesses
  1. The components and records that determine the central empirical comparison are insufficiently specified or available for inspection. Section 3 and Appendix A.7 explicitly state that the available record lacks raw scorecards, learned output texts, fitted weights, and the construction, target-building, estimation, and decoding implementations. Most consequentially, the exact adapted CCL serialization cannot be independently inspected, although its representation cost is central to the strongest baseline’s reported advantage.
  2. The paper does not establish whether the principal CCL-Min-versus-trace ordering remains when explicit equation overlap is separated from entailed-value and difference agreement, or when logical equivalence is measured directly in this exact affine domain. How much of the reported advantage reflects preserved information rather than agreement with a particular signature?
  3. The consumer evaluation provides limited evidence about the relationship between source fidelity and practical recover ability.The headline closure results use the byte axis on the full report set, whereas consumer recovery pools both budget axes on a smaller subset. The study does not yet establish whether representations with higher source agreement are easier or harder for a consumer to use under matched conditions.
Questions for authors
  1. Can you provide complete specifications or executable implementations of the adapted CCL-Min/CCL-Core serializations, trace decoder, and target construction, together with representative source texts, raw learned outputs, decoded equation sets, and scores linked to their checkpoints? These are needed to assess what determines the central format comparison.
  2. Does the CCL-Min-versus-trace ordering persist when explicit general equations are reported separately from entailed values and pairwise differences? What fraction of decoded outputs is logically equivalent to the source equation system?
  3. Are the matching 1B and 2.8B byte axis results identical only at the displayed precision, at the aggregate level, or at the per-record level? Can independently generated outputs and checkpoint identifiers verify their attribution?
6 comments

Thanks for the review:

  1. Signature Checks on the Byte and Token Axes (Sections 2 and 5, Appendix B) On the byte axis, CCL-Min leads every trace translator under all five scoring rules—by +0.189 under the paper's signature (a), +0.193 under consequences only (b), +0.085 under explicit equations only (c), +0.360 under exact affine equivalence (d), and +0.124 under values only (e) (gaps are computed from unrounded AUCs)—and for every encoder seed (0.861, 0.861, 0.857 under (a); smallest margin over any trace seed across all rules is +0.083). The explicit members' contribution (a) − (b) to the CCL-Min − TE, 1B gap is −0.0045 on bytes and −0.0070 on tokens: they do not create the gap.

We re-scored all 236,800 deterministic-decoder report records under five rules in Appendix B (Signature decomposition across five scoring rules): (a) the paper's signature, (b) consequences only (values and pairwise differences; first-run predicates missed non-unit values or differences in 186 decoded records, moving no AUC by more than 4.8e-06), (c) explicit equations only (Jaccard, computed whether or not the decoded system is consistent), (d) exact affine equivalence (1 or 0 per record via RREF row-space equality), and (e) values only: (Appendix B reports all 13 systems, including TE, 1.4B, 410M, 160M, 70M, and Qwen3 0.6B and 1.7B.)

Token-axis ordering and seed dependence: On the token axis, Canonical JSON leads all learned systems under (a), (b), (c), and (e), while Oracle core leads under (d) (0.202; CCL-Min 0.196, Canonical JSON 0.190). Between CCL-Min and the best trace (TE, 2.8B), CCL-Min leads under (a) (+0.057), (b) (+0.064), and (d) (+0.196), whereas four traces (TE, 2.8B; TE, 1B; TE, Qwen3 0.6B; TE, Qwen3 1.7B) lead CCL-Min under (c) (−0.020: 0.310 vs 0.330) and (e) (−0.033: 0.498 vs 0.531). Per-seed results in Appendix B show that CCL-Min seed A stays within token budgets on 769 / 770 / 724 / 800 of 800 records (AUC 0.561 under (a), 0.682 under (e)), whereas seeds B (0.364 / 0.412) and C (0.352 / 0.400) exceed the token budget on 55.8 %–74.1 % of records at the three smaller allowances; 670 of CCL-Min's 3,134 over-budget token records are complete, exact codes (Appendix F).

2.Logical Equivalence by Budget, Oracle-Core Correction, and Regression Check

Correction in submitted Table 5 and regression check: We correct one scoring error in submitted Table 5's Oracle core byte row from 0.000 / 0.725 / 1.000 / 1.000 (AUC 0.768) to 0.000 / 1.000 / 1.000 / 1.000 (AUC 0.850), caused by the submitted scorer applying the byte penalty to a JSON decoding wrapper on 220 of 800 compliant records at byte budget 0.45; the token row (0.202) and all translator and structured-control rows are unchanged. Separately, re-scoring all 236,800 decoded records reproduces the recorded per-record unpenalized scores, statuses, and budget flags on 0 records with a difference, matching 25 of 26 series × axis AUCs (largest difference 3.7e-15) and 33 of 34 rows of submitted Tables 5, 6, and 8.

  1. For Weakness 1, Question 1, and Question 3: the specifications, implementations, and checkpoint-linked records in the revised supplementary archive, the representation cost of each format, and the per-record comparison and attribution of the 1B and 2.8B translators.

The revised supplementary archive provides each item that submitted Appendix A.7 listed as absent, except the fitted adapter weight tarballs, for which the 39-cell hash table is included:

Adapted CCL serialization and format grammars (Appendix K, Code format specifications, decoders, and target construction, and docs/CODE_FORMATS.md): CCL-Min serializes canonical equations on one line as @CCL/1m E=<eq>|<eq>|...; CCL-Core uses header @CCL/1, TYPE=affine-rational-v1, and EQ=<eq> lines. Decoder, target-builder, construction, and estimator implementations (code/frozen/, code/revision/, code/rebuttal/): executable decoders, scorecard/estimator module, exact rational closure oracle, target construction, world construction, trace checkers, consumer evaluation and parser code, re-scoring scripts, and the scorer revision patch. Raw learned outputs, source texts, decoded sets, scorecards, and checkpoint table (data/, results/, consumer_journals/, Appendices A.7 and D): all 268,800 report records and strong-encoder records with source texts, reference equations, requested entities, answers, raw outputs, 236,800 decoded equation sets, consumer logs, and per-record scorecards keyed by cell hash and selected checkpoint to the 39-cell hash table. The adapter weights are not included; their 39-cell hash table is included. Format representation cost (Appendix B, Format representation cost): measured as 1 minus the closure agreement of the largest budget-fitting prefix of each format's canonical serialization across the 800 report items (a canonical-prefix baseline) at budgets 0.30 / 0.45 / 0.60 / 0.80 (and cost-AUC): CCL-Min: bytes 0.258 / 0.000 / 0.000 / 0.000 (cost-AUC 0.039); tokens 0.805 / 0.593 / 0.292 / 0.000 (cost-AUC 0.401). CCL-Core: bytes 0.533 / 0.000 / 0.000 / 0.000 (cost-AUC 0.080); tokens 0.859 / 0.683 / 0.432 / 0.037 (cost-AUC 0.492). Canonical JSON: bytes 0.624 / 0.072 / 0.000 / 0.000 (cost-AUC 0.115); tokens 0.841 / 0.644 / 0.394 / 0.000 (cost-AUC 0.457). Structured prose: bytes 0.630 / 0.218 / 0.000 / 0.000 (cost-AUC 0.160); tokens 0.854 / 0.697 / 0.498 / 0.195 (cost-AUC 0.551). Trace (TE): bytes 0.644 / 0.213 / 0.000 / 0.000 (cost-AUC 0.160); tokens 0.910 / 0.776 / 0.587 / 0.218 (cost-AUC 0.618).

Reviewer_4McaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper establishes a verifiable domain for text-to-text compression: affine arithmetic statements. They show that while models compress with internal consistency, the facts themselves are often corrupted in the process. They show across various models and code formats and compression ratios how well the facts are recovered.

The paper seems overall experimentally sound, it defines verifiable tasks and solves them, ablating effects of different parameters on the reconstruction task they pose. However, I think there is a lack of motivation for someone like myself not working directly in this area. I would like to see more notes on the potential applications of the work to justify the extensive, bespoke, task setup that has been created for this paper. My understanding is that the central application is in quantifying the quality of text-to-text compression. The task itself feels a bit narrow to be widely applied to other domains. The density of information in this paper makes it hard to know where to focus attention. More details on the compression code formats would be appreciated.

Critical strengths

The arithmetic domain allows the paper to reliably quantify translation agreement and give definite results. The paper does a rigorous ablation across different settings making the results trustworthy and the evaluations sound.

Critical weaknesses

It often feels that the findings on the arithmetic domain do not have wider applicability, making the paper less interesting than it could otherwise be.

Questions for authors

How do you forsee the your results being used by other researchers or practitioners?

Comments on the paper

1 comment

We thank all three reviewers for their constructive reviews. The revised manuscript and supplementary archive incorporate four additions addressing the shared questions across reviews:

  1. Supplementary Release and Format Specifications (Section 3, Appendices A.7, D, K, and Supplementary Material) The revised supplementary archive and manuscript add the items listed as absent in submitted Appendix A.7:

a. Format specifications and target construction (Appendix K, Code format specifications, decoders, and target construction, and supplementary docs/CODE_FORMATS.md): grammars, canonicalization rules, deterministic decoders, target construction, and worked examples for CCL-Min, CCL-Core, Canonical JSON, Structured prose, and Trace code (TE).

b. Raw outputs, source texts, decoded sets, and scorecards (supplementary data/): all 268,800 report records and strong-encoder records with their source texts, reference equations, requested entities, answers, raw outputs, and scorecards keyed by cell hash and selected checkpoint, plus all 236,800 decoded equation sets.

c. Implementations (supplementary code/): the decoders, scorecard, rational closure oracle, target and world construction, the three trace checkers, the evaluation module, the scorer revision patch, and the consumer and parser code; the supplementary MANIFEST.md lists the submitted and re-run consumer logs with their sizes and SHA-256 hashes.

d. Checkpoint hash table (Appendices A.7, D, and supplementary results/): all 39 trained cells with base model, revision, parameters, seed (A, B, C), selected checkpoint, and cell, adapter, and tarball SHA-256 hashes and byte sizes. The adapter weights are not included; their 39-cell hash table is included.

  1. Five-Rule Signature Decomposition and Oracle-Core Correction (Sections 2 and 5, Appendix B) Re-scoring all 236,800 deterministic-decoder records under five rules in Appendix B (Signature decomposition across five scoring rules)—(a) paper signature, (b) consequences only, (c) explicit equations only, (d) exact affine equivalence, and (e) values only—shows:

a.Byte axis: CCL-Min leads every trace translator under all five rules—0.860 vs 0.672 under (a), 0.855 vs 0.662 under (b), 0.564 vs 0.480 under (c), 0.647 vs 0.287 under (d), and 0.906 vs 0.782 under (e)—and for every encoder seed (0.857–0.861 under (a)). The explicit members' contribution (a) − (b) to the CCL-Min − TE, 1B gap is −0.0045 on bytes and −0.0070 on tokens: they do not create the gap.

b. Token axis: Canonical JSON leads all learned systems under (a) (0.514), (b) (0.500), (c) (0.404), and (e) (0.649), while Oracle core leads under (d) (0.202; CCL-Min 0.196, Canonical JSON 0.190). Four traces lead CCL-Min under (c) (0.330 vs 0.310) and (e) (0.531 vs 0.498), and CCL-Min's token result depends on its encoder seed through budget compliance (seed AUCs 0.561 / 0.364 / 0.352 under (a); Appendix B).

c. Correction in submitted Table 5 (Section 5, Appendices A.7 and B): we correct one scoring error in the submitted Table 5 Oracle core byte row from 0.000 / 0.725 / 1.000 / 1.000 (AUC 0.768) to 0.000 / 1.000 / 1.000 / 1.000 (AUC 0.850). Cause: the submitted scorer wrapped the oracle's code in the JSON schema for decoding, applied the byte penalty to the wrapped length, and flagged compliance from the raw length, penalizing 220 of 800 compliant records at budget 0.45. The token axis (0.202) and all translator and structured-control rows are unchanged.

d. Separately, re-scoring all 236,800 decoded records reproduces the recorded per-record unpenalized scores, statuses, and budget flags on 0 records with a difference, matching 25 of 26 series × axis closure-AUCs (largest difference 3.7e-15) and 33 of 34 rows of submitted Tables 5, 6, and 8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.