- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HiContext compacts an agent's history into trajectory, subtask, and state blocks. A separate LLM chooses the level every 30K new tokens and rewrites part of the context, while earlier blocks are mostly kept. The task agent is Qwen3.5-27B and the compactor is Qwen3.5-9B. The paper evaluates on BrowseComp-Plus and NL2Repo-Bench, with fixed-level ablations and a teacher fine-tuning experiment.
Critical strengths
-
The method is simple to deploy. It does not require retraining the task model. The prompts, budget rules, and task-model versus compactor costs are well documented, which makes the system easier to reproduce.
-
The gains appear in both domains, on BrowseComp-Plus and NL2Repo-Bench. It also beats subtask-only compaction at the same context length on BrowseComp-Plus. This suggests that the gain is not only from using less context.
Critical weaknesses
-
It is hard to separate hierarchy from prompt engineering. HiContext has detailed domain-specific instructions to preserve document IDs, start.md, paths, and errors. Some baselines lose exactly this information or suffer from tool errors. The fixed-level ablations do not include an equally engineered single-level summarizer or a simple level-selection control
-
The statistical evidence is thin. There are no repeated-run confidence intervals or significance tests. The NL2Repo ablation uses a small amount of instances, and the fine-tuning result is a single in-domain comparison
Questions for authors
-
How many independent tasks and runs support each result, and what does “Score” mean on each benchmark? Can you add per-task confidence intervals
-
Can you compare against a single-level summarizer with the same preservation instructions and a simple random or count-based level-selection rule
-
How many training examples survive the 0.8 outcome filter, and was that filter ablated? Please also clarify whether any question can appear in both training and evaluation after concatenation
3 comments
Trained Compactor Transfer (W2)
We also evaluate on the shopping task of WebArena-Verified Hard, with Qwen3.5-27B as task model and the BrowseComp-Plus-trained compactor.
Transfer to WebArena-Verified Hard shopping tasks:
| Method | Pass Rate | Cost / Task |
|---|---|---|
| Sliding Window | 41.07% | $0.094 |
| Context Folding | 46.43% | $0.122 |
| HiContext (Qwen3.5-9B) | 48.21% | $0.058 |
| HiContext (BC+ SFT Qwen3.5-9B) | 51.79% | $0.071 |
These results show that HiContext outperforms the baselines on this additional benchmark and that the BrowseComp-Plus-trained compactor improves performance on an unseen benchmark.
Training Filter and Data Split (Q3)
We split original BrowseComp-Plus questions before concatenating them and generated training rollouts only from the training split. Consequently, concatenation cannot cause an original question to appear in both training and evaluation. Across four rollout rounds, the 0.8 outcome filter retained 4,212 compaction examples. This use of task outcomes to filter training data is consistent with AgentTuning (Zeng et al., 2024). We did not ablate the filter or its threshold.
We hope these clarifications address the concerns and would greatly appreciate your reconsideration of the current assessment based on our responses. We also welcome further discussion and are happy to provide additional analysis if helpful.
Dear Reviewer azBD,
Thank you for your time and effort in reviewing this paper. We would appreciate it if you could review our rebuttal and check whether your concerns are addressed. Thanks again for your time and consideration!
Regards from authors.
We thank you for recognizing HiContext’s design and its improvements on both benchmarks. We would like to address your concerns as follows:
Prompt Engineering and Hierarchical Compaction (W1)
In the original experiments, HiContext and its fixed-level variants used the domain-specific stall footer; the external baselines used their original prompts and received no footer. The fixed-level variants also shared HiContext’s preservation instructions for their assigned level, so this ablation supports the benefits of hierarchical compaction. We retained each external baseline’s original prompt to preserve the operations it was designed to support.
For the context engineering ablation, we rerun the compaction baselines with the same domain-specific footer appended to their compaction outputs and add preservation instructions to their original prompts. We add state-level instructions to ReSum for its consolidated task summary, subtask-level instructions to Context-Folding for its subtask-based folding, and trajectory-level instructions to AgentFold to preserve concrete execution details during its folding at every turn.
Due to time constraints during rebuttal, the additional NL2Repo-Bench experiments labeled "30 tasks" use 30 randomly sampled instances. We will extend these evaluations to all 104 instances in the camera-ready version.
Baseline scores before and after adding preservation instructions and footers:
HiContext still outperforms augmented baselines by 10.0 percentage points on BrowseComp-Plus and 3.07 percentage points on NL2Repo-Bench. These results support the benefit of hierarchical compaction beyond the engineered prompt.
Single Level and Random Level Summarization (Q2)
The fixed-level ablation in Section 4.3 already provides the single-level summarization comparison you requested. Each uses the same instructions as HiContext for its assigned level and always compacts at that level.
With the rest of HiContext configuration held fixed, random level selection scores 60.7 versus 65.3 for HiContext on BrowseComp-Plus and 31.19 versus 34.75 on NL2Repo-Bench (30 tasks). This comparison shows that the level-selection policy contributes to performance.
Statistical Significance and Full NL2Repo Ablation (W2 + Q1)
BrowseComp-Plus groups 150 test questions into 30 five-question tasks. Score is the mean fraction of questions answered correctly per task. NL2Repo-Bench has 104 repository-generation tasks. Score is the mean fraction of passing test cases per task. Both scores are reported as percentages.
We supplement the evaluation with three independent runs, paired confidence intervals, and an expanded ablation on the full NL2Repo-Bench evaluation set.
Repeated-run results: We report the mean ± standard deviation across three independent runs.
Paired uncertainty analysis: We conduct paired bootstrap analyses and compute 95% confidence intervals using the percentile method to quantify uncertainty in the performance differences between methods.
All four confidence intervals exclude zero, providing statistical evidence for positive improvements in these comparisons.
We also expand the NL2Repo-Bench ablation from the original 40-task subset to all 104 tasks.
Full NL2Repo-Bench ablation: