- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies what learned text compression preserves in exactly solvable affine arithmetic worlds. A construction record projects each source into canonical rational equations, and a fixed decoder maps compressed code into the same language. The evaluation signature contains explicit equations, uniquely determined variable values, and pairwise differences; closure agreement is the Jaccard overlap between source and code signatures. Grammar and internal-constraint acceptance, source agreement, and another model’s value recovery are measured separately. Over-budget outputs remain in the denominator and receive zero budgeted agreement.
Training uses 24,000 worlds, with 800 renderings of 200 held-out worlds and a consumer subset of 240 renderings from 60 worlds. Renderings of the same world remain in one split. LoRA trace translators span Pythia 70M to 2.8B and two Qwen models with three seeds. JSON, CCL, and template-prose controls also have trained encoders; a separate oracle reads the construction directly. Byte and reference-token budgets generate separate outputs at four allowances. CCL-Min achieves byte AUC 0.860 versus the best trace’s 0.672; on the token axis JSON reaches 0.514 and CCL-Min 0.426. Scale findings concern the shared training procedure, while consumer findings depend on reader and decoding conditions. Additional analyses cover redundant renderings, sampled failures, parsing, and stop reasons.
Critical strengths
-
The source equations and fixed decoder make omissions, changed facts, and derivable redundancy inspectable. In Table 1, the 1B, 2.8B, and Qwen 0.6B failure samples have no budget violations but contain checker-accepted outputs, directly establishing learned source errors that internal checks miss. World-level split isolation and inclusion of over-budget outputs also make the controlled comparison easier to interpret.
-
JSON and CCL controls use trained encoders, while the oracle separately represents direct access to correct facts. Byte and token conditions generate their own budgeted outputs, and Table 2 shows that the leading format changes with the cost unit. Multiple seeds, scales, and a second model family add useful checks. The paper also distinguishes sampled failures from prevalence estimates and an AUC plateau from model equivalence, while keeping consumer conclusions tied to the stated response conditions.
Critical weaknesses
-
The signature on pages 2 and 3 includes explicit equations, so a fully equivalent rewriting can lose agreement. Replacing x=2, y=x+3, z=2y with x=2, y=5, z=10 preserves all affine information but loses the explicit member 2y-z=0 under the chosen rules, scoring 6/7. This representation dependence is disclosed, but the main CCL versus trace gap is not checked under equivalent bases, value-and-difference-only scoring, or alternative consequence weights. The 0.860 versus 0.672 result therefore first establishes an advantage for this signature; its interpretation as general source-information preservation needs further validation.
-
Appendix A.7 on page 16 explicitly states that the current record omits the adapted CCL serialization, decoder, target builder, construction implementations, and raw learned outputs. The descriptions do not suffice to independently reconstruct the key formats. Format overhead, translation errors, omissions, and budget violations all lower AUC. Appendix F’s error labels partly derive from already budget-penalized scores, so they do not adequately separate these causes. This limits both the explanation of which design improves performance and the inspectability of the central measurement study.
-
The consumer comparison on page 8 pools two budget axes and a smaller world subset, whereas the main closure comparison uses the full report set’s byte axis. Their orderings therefore do not directly show preservation versus readability for the same codes. The paper acknowledges this and already reports parsing, stop reasons, and a lower-reasoning GPT-OSS condition. However, substantial truncation remains under the 256-token cap. Missing information and consumer decoding failures have not been separated on matched codes and worlds, limiting guidance for practical compression choices.
Questions for authors
-
How do the CCL versus trace ranking and gap change under value-and-difference-only scoring, equivalent equation bases, and alternative consequence weights? Can the contribution of explicit-equation members to the AUC gap be quantified?
-
Please specify CCL syntax, decoding, and target construction sufficiently for independent implementation, and use raw outputs to distinguish representation cost, translation errors, omissions, and budget violations. Can output-to-checkpoint mapping also be verified?
-
On the same codes and worlds, which format differences persist when source agreement and consumer recovery are reported together and GPT-OSS has enough output space to finish? Can the existing parsing and stop statistics stratify that comparison?
6 comments
Five-Rule Closure-AUC Across All 236,800 Decoded Records (Appendix B)
Token-axis ranking and encoder-seed compliance: On tokens, Canonical JSON leads all learned systems under (a), (b), (c), and (e), while Oracle core leads under (d) (0.202; CCL-Min 0.196, Canonical JSON 0.190). Between CCL-Min and TE, 2.8B, CCL-Min leads under (a) (+0.057), (b) (+0.064), and (d) (+0.196), whereas four traces (TE, 2.8B; TE, 1B; TE, Qwen3 0.6B; TE, Qwen3 1.7B) lead CCL-Min under (c) (−0.020) and (e) (−0.033). Per-seed results (Appendix B) show that CCL-Min seed A stays within token budgets on 769 / 770 / 724 / 800 of 800 records (AUC 0.561 under (a), 0.682 under (e)), whereas seeds B (0.364 / 0.412) and C (0.352 / 0.400) exceed the token budget on 55.8 %–74.1 % of records at 0.30–0.60. Correction in submitted Table 5 and regression check: We correct one scoring error in submitted Table 5's Oracle core byte row from 0.000 / 0.725 / 1.000 / 1.000 (AUC 0.768) to 0.000 / 1.000 / 1.000 / 1.000 (AUC 0.850), caused by the submitted scorer applying the byte penalty to a JSON decoding wrapper on 220 of 800 compliant records at byte budget 0.45; the token row (0.202) and all translator and structured-control rows are unchanged. Separately, re-scoring all 236,800 decoded records matches the recorded per-record unpenalized scores, statuses, and budget flags on 0 records with a difference, reproducing 25 of 26 series × axis AUCs (largest difference 3.7e-15) and 33 of 34 rows of submitted Tables 5, 6, and 8.
For Weakness 2 and Question 2: specifying CCL syntax, decoding, and target construction, verifying output-to-checkpoint mapping, and using raw outputs to separate representation cost, translation errors, omissions, and budget violations.
Specifications, Implementations, and Output-to-Checkpoint Mapping (Appendices A.7, D, K, Supplementary Material)
Syntax, decoders, and target construction (Appendix K, Code format specifications, decoders, and target construction, and supplementary docs/CODE_FORMATS.md): CCL-Min serializes canonical equations on one line as @CCL/1m E=<eq>|<eq>|...; CCL-Core uses header @CCL/1, TYPE=affine-rational-v1, and EQ=<eq> lines. Target construction orders a world's canonical equations and takes the largest prefix fitting the byte or token budget. The supplementary archive provides executable decoders, target and world construction, the rational closure oracle, the three trace checkers, the scorer revision patch, and all 268,800 report records (and strong-encoder records) with source texts, reference equations, raw outputs, 236,800 decoded equation sets, and scorecards. Output-to-checkpoint mapping (Appendices A.7 and D, supplementary results/): Every one of the 249,600 learned records maps with 0 checkpoint mismatches and 0 system-id mismatches to one of the 39 trained cells, which have 39 / 39 / 39 of 39 distinct cell, adapter, and adapter-tarball SHA-256 hashes. The adapter weights are not included; their 39-cell hash table is included. This checks every record's labels against the cell table, and the stored objects' size and checksum against the manifest; for Pythia 1B seed A (cell 7287B0B85BDC, checkpoint-376) and Pythia 2.8B seed A (cell 1D312290FEBA, checkpoint-376), each system's complete report grid was regenerated greedily from the base model and the cell's adapter weights, with the same generation code and batching as the report run, and 12,800 of 12,800 regenerated outputs are byte-identical to the recorded raw outputs (800 of 800 at byte budget 0.45 for each system; 6,400 of 6,400 per system over 8 budget cells), with 0 mismatches.
Thanks for the for the detailed analysis and concrete checks.
Signature Decomposition and Worked Example Across Five Scoring Rules (Sections 2 and 5, Appendix B) On the byte axis, CCL-Min leads the best trace translator (TE, 2.8B = TE, 1B) under all five scoring rules—by +0.189 under the paper's signature (a), +0.193 under consequences only (b), +0.085 under explicit equations only (c), +0.360 under exact affine equivalence (d), and +0.124 under values only (e) (gaps are computed from unrounded AUCs)—and for every encoder seed (0.857–0.861 under (a)). The explicit members' contribution (a) − (b) to the CCL-Min − TE, 1B gap is −0.0045 on bytes and −0.0070 on tokens: they do not create the gap. The alternative consequence weightings evaluated are (b), (d), and (e); no other weighting was run.
We ran your value-basis rewriting (x=2, y=5, z=10 against x=2, y=x+3, z=2*y) and the Section 2 fixtures through the exact rational oracle under five rules: (a) paper signature, (b) consequences only (values and pairwise differences; first-run predicates missed non-unit values or differences in 186 decoded records, moving no AUC by more than 4.8e-06), (c) explicit equations only (Jaccard, computed whether or not the decoded system is consistent), (d) exact affine equivalence (1 or 0 per record via RREF row-space equality), and (e) values only: row 142 - 149 in the revised version