acceptodds
Withdrawn from ICLR 2027

Task-Aligned Context Distillation: Set-Wise Selection with Structural and Geometric Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-context distillation aims to compress noisy inputs into minimal sufficient subsets while preserving downstream reasoning performance. Existing methods fall into two paradigms: point-wise methods score context units without modeling inter-chunk dependencies, while heuristic set-wise methods optimize diversity without grounding in solver feedback. Both are susceptible to semantic collapse, structure blindness, or solver mismatch under strict compression budgets. We reframe context distillation as a **solver-aligned set-wise** optimization problem, and introduce **TaCo**, a solver-in-the-loop policy learning framework. Solving this formulation via naive reinforcement learning (RL) exposes a fundamental challenge: continuous embeddings provide no inductive bias, yielding discontinuous reward surface and high reward variance that renders credit assignment intractable. By identifying two orthogonal sources of reward instability, TaCo overcomes this with two complementary priors: a **structural prior** preserves the internal integrity of each context unit, and a **geometric prior** promotes the diversity across selected units. Together, these priors substantially reduce the effective RL search space, enabling stable credit assignment under sparse solver feedback. On academic long-context benchmarks, TaCo outperforms SOTA compressors on unstructured text and even exceeds the full-context baseline on structured text. On an industrial benchmark with extreme length, high noise and mixed structure, TaCo achieves 98% and 102% of full-context F1 at 2.18% and 9.71% token retention respectively. Beyond aggregate performance, mechanistic diagnostics confirm that TaCo's three core components—the structural prior, geometric prior, and solver-aligned policy—orthogonally mitigate all three failure modes, validating the representational design of the framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Task-Aligned, Context Distillation, Set-Wise Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mabdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In general, this paper want to address some problems of long-context distillation. Specifically, it frames context compression as a subset selection problem—constrained by a token budget and aligned with a solver—and introduces TaCo, a method that employs REINFORCE to train the selection policy while incorporating structural and geometric priors as features within the state representation. Experiments conducted across multiple benchmarks demonstrate some performance improvements, and ablation studies illustrate the impact of the various components.

Critical strengths

The problem addressed in this paper have some practical significance, as long-context tasks are indeed common in real-world scenarios (and thus may attract some interest from the community). Furthermore, the paper candidly discusses certain limitations.

Critical weaknesses

In fact, I am a bit confused or seriously concerned about some of the experimental results and data presented in this paper.

For instance, the "Task Score" for the "w/o Symbolic" ablation in Table 4 appears identical to that of the "Dense Top-K" baseline in Table 2 across multiple benchmarks. However, there is no mention of their equivalence (that's why I am confused); furthermore, the results for the two differ on the Financial-250k dataset. I thus suggest that the authors should provide a thorough explanation and analysis regarding these results.

Additionally, regarding the latency comparison (the "Com" column) for TaCo in Table 11—specifically the values ​​0.12, 0.24, 0.36, 0.48 (i.e., 0.12*n), etc.—there is a consistent increment of 0.12 between each interval. Is there a specific pattern behind this? The authors should also clarify this point.

Questions for authors

See Critical Weaknesses and,

  1. How were the results in Table 11 obtained?

  2. The values ​​for "w/o Symbolic" in Table 4 and "Dense Top-K" in Table 2 appear to be identical; do they represent the same method?

Reviewer_BuYVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies efficient long-context compression, focusing on three limitations of existing methods: Semantic Collapse, where redundant content is selected while complementary information is missed; Structure Blindness, where information structure is damaged during compression; and Solver Mismatch, where selected content is not aligned with the downstream solver's needs. To address these problems, the authors propose TaCo, an RL-based context subset selector trained using feedback from a frozen downstream solver. Its policy uses additional state features derived from structural and geometric priors. The paper reports competitive results in cross-domain zero-shot transfer and extreme industrial compression, together with component ablations and training analyses.

Critical strengths

The paper uses downstream task performance as feedback to train the context selector, directly connecting compression decisions to the final task. The reported results show that a selector trained on industrial data can achieve competitive performance on academic benchmarks without further training, while substantially reducing input length. This provides useful empirical support for the practical value of the approach.

Critical weaknesses

1.In the paper, key concepts are used without sufficient explanation, and important claims are introduced without clearly establishing their connection to the preceding discussion. For instance, Section 3.1 uses "intra-unit discrete boundaries" without clearly defining their relationship to the context units, while structural and semantic information are described as "two orthogonal dimensions" without specifying what "orthogonal" means in this setting. These concepts are then used to motivate the proposed method, leaving readers to infer essential connections themselves. This makes the paper difficult to follow even after careful reading. 2.The motivation for the two priors relies on two claims that are not adequately supported. In Section 3.1, the paper asserts that Structure Blindness leads to a discontinuous reward surface, but nearby representations producing substantially different rewards do not, by themselves, establish discontinuity. Similarly, the low additional utility of redundant selections does not imply the high reward variance attributed to Semantic Collapse. The theoretical analysis in Appendix B.1 also contains a gap: the proof of Corollary B.1 applies the law of total variance while omitting the variance of the conditional mean, without establishing that this term vanishes. Consequently, the claimed variance reduction does not follow from the proof as written. These issues undermine the stated rationale for introducing the two priors.

Questions for authors

The paper proposes structural and geometric priors to address Structure Blindness and Semantic Collapse, respectively. However, the structural prior adds structure-related features while the selector still keeps or discards entire context units. How do these features help prevent the structural damage described in the paper? Similarly, the geometric prior provides features based on semantic similarity, which does not necessarily imply information redundancy. How does this support the claimed mitigation of Semantic Collapse, particularly the preservation of complementary information?

Reviewer_oj5BOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors treat long-context compression as "set selection aligned with the downstream solver." There are two reasons: pointwise scorers cannot express the utility arising from set composition, and heuristic set rules cannot be aligned with a specific solver; these two points are supported by Lemma A.1 and A.2, respectively. The TaCo method selects unit by unit using sequential REINFORCE, and the state includes, in addition to the raw embeddings and the query, two kinds of priors. One is structural features, obtained by having a frozen CodeLlama-7B write code once for each schema; the other is geometric features, using cached similarities to k-means centroids and medoids. A running summary of the selected items is also part of the state. The reward mixes task score, embedding consistency, and compression rate. The training data consists of about 35k trajectories from private financial logs, and the solver is GPT-4.1.

Critical strengths

The ablation is the part I found most convincing. Removing the symbolic module hurts RepoBench badly — CER drops from 95.7 to 64.6, and the task score from 72.5 to 56.4 — while the cosine distance barely moves; removing the manifold module does the reverse. That is a clean dissociation, and the regex/TF-IDF controls point the same way. I also appreciate that the cross-solver table (Table 12) is reported even though the F1 gain doesn't transfer: GPT-4.1 improves, DeepSeek-V3.1 and Gemini-2.5-Pro do not. The transfer to NarrativeQA/GovReport/RepoBench/LCC is a real distribution shift, not an in-distribution test.

Critical weaknesses
  1. A reference implementation is provided, and the main results are averaged over 5 runs, which is good. But the corpus is private and the GPT-4.1 version isn't specified, so I can only check so much. I understand the data can't be released; my concern is which claims are independently verifiable, not the decision to keep it private. On LongBench the gains over the strongest baseline are modest — HotpotQA is 0.531 vs 0.494, with full context at 0.546; QMSum is essentially a tie. On Financial-250k, TaCo Extreme gets a high headline score at 2.18% retention, but its F1 is below several baselines, and no baseline was run at that budget. The 102% figure also uses a sliding-window full-context reference, which is disclosed in Section 4.1 but still matters. On RepoBench, TaCo beats full context, which the "code completion needs little context" explanation doesn't really explain.
  2. The Table 4 rows still bother me. The four task scores in the w/o Symbolic row are identical to Dense Top-K in Table 2, even though w/o Symbolic should keep the manifold module and the policy. The MMR (F1-Max) row also matches BGE-M3+Schema+MMR, but that one is expected from the caption. I can't tell from the paper whether the w/o Symbolic row was re-run or is the Dense Top-K result; the scores are identical. On Financial-250k the picture is different — w/o Symbolic mostly just keeps more tokens — so the academic table is the odd one. I'd also like a variant that drops the priors but keeps the policy, since the current w/o RL Policy conflates the two. The limitations section is mostly qualitative, which is fine, but it doesn't answer this.
Questions for authors
  1. Was the w/o Symbolic row run separately, with the manifold module and policy intact, or is it the Dense Top-K result? The task scores are identical, so I can't tell. I understand the MMR (F1-Max) row matching BGE-M3+Schema+MMR is by construction.

  2. Since the corpus isn't released, was the de-identification audited by anyone outside the project? I've noted this under ethics rather than as a technical objection.

  3. At minimum, Rule-Based and LLMLingua2 at ~2.2% retention would give the 98% claim a fair comparator; Dense Top-K would be a useful third. If one of them can't run at that budget, saying so is fine. The non-overlapping full-context pass is optional — useful for the 102% figure, but I wouldn't make it a condition.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.