acceptodds
Withdrawn from ICLR 2027

Evaluating Relational Context Compression at Realized Token Budgets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Context compression must preserve answerable information within a token budget, yet shorter text alone cannot isolate the effect of its representation. We study Telegraph English, which rewrites retrieved passages into relational statements, using a pre-registered encoder–consumer matrix and an open compiler trained by distillation. Six of seven encoders violated the registered output schema on most or all inputs. Only 33 of 9,600 budgeted outputs fell within the registered band: at or below the requested budget and no more than two percent short. These findings motivate evaluating answer F1 at realized token ratios. The matrix was collected, but all registered confirmatory comparisons were unavailable because their comparator or calibration conditions were missing or ineligible. The remaining answer-F1 comparisons with full prose are descriptive and score construction failures as empty answers. The second compiler failed its pre-registered acceptance test against a budget-matched pruning baseline. Its out-of-distribution component was unscorable because answer keys were absent, and citation-reach judging was not reached. Context-compression comparisons require measured token budgets, explicit construction failures, and available comparator evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
context compression, prompt compression, retrieval-augmented generation, multi-hop question answering, evaluation methodology, pre-registration, distillation, small language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_LrjxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates Telegraph English (TE), a context-compression format that rewrites retrieved passages as compact relational statements (e.g., "Orin | adviser | Sana"). It reports two studies: a pre-registered encoder–consumer matrix and a distilled compiler.

Critical strengths

None

Critical weaknesses
  1. The paper is poorly written. The authors assume readers can understand a lot of complicated knowledge and jargons. For example, right in introduction, they introduce "reader’s evidence and its token cost". What is reader here? Or right after it, "Only 33 of 9,600 outputs in our budgeted generation study fall within the registered 98–100% target band. The two anchors’ median lengths exceed their requests throughout the measured grid." How are readers supposed to understand what outputs are these? Or which "registered 98–100% target band" are? The paper needs a substantial rewriting before it's ready for another submission. After reading intro section, I still don't get what the main contribution of the paper is.
  2. At the end, the paper is an evaluation paper, yet it's very hard to understand what the evaluation wants to deliver as result and recommendation. The paper's central scientific question goes unanswered, which leaves very little to evaluate as a contribution.
  3. The headline "failure" findings are largely by design and are not tested against simple fixes. Hitting 98–100% of a target length through free-form generation with a single count-feedback revision is a very strict criterion. A 33/9,600 hit rate is unsurprising and says little about relational compression itself.
Questions for authors

Please see weaknesses.

Reviewer_CpqXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This submission reads as an incomplete study: the main controlled comparisons were not completed, and several planned evaluations remain unavailable. It reports budget-control failures in Telegraph English compression and poor performance from a distilled compiler.

Critical strengths
  1. Measuring realized token budgets is important for fair compression comparisons.
  2. The authors clearly disclose failed experiments and missing evaluations.
Critical weaknesses

The reporting is transparent, but the missing experiments and limited analysis leave the study insufficiently developed for acceptance.

  1. Section 4 states that all registered confirmatory comparisons are unavailable, leaving the central representation comparison unresolved.
  2. The compiler comparison does not verify matched realized budgets or adequately explain the observed failure (Section 5).
  3. Raw examples, predictions, and trained weights are unavailable, limiting independent verification.
Questions for authors

See Weaknesses.

Reviewer_riuPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates Telegraph English in multiple aspects.

Critical strengths

The figures and tables are clear and informative.

Critical weaknesses

This paper evaluates a specific method, Telegraph English, but not explain whether the relational context compression has any other related works. It seems only Telegraph English uses it, which makes the findings of this paper hard to generalize to the broader AI community. And these is no baseline compression method in the evaluation, making the evaluation not solid.

Questions for authors

Could you explain the broader benefit of the proposed study to the AI community?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.