- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compresses older tool observations of SWE agents into soft tokens (LOHA). Agent turns and the last K observations are kept as raw text. Training (ACD) uses a low-rank patch with two KL terms: one distills the base model's full-text predictions into the compressed view, and one anchors the model's full-text behavior to the original model. On SWE-bench Verified with two 4B models, K=3 cuts context per call by 43–57%. Resolve rate drops from 14.5% to 12.1% (Qwen3-4B) and from 27.5% to 21.8% (SWE-Master-4B). Additional results cover a 32K context limit, a K sweep, and serving throughput.
Critical strengths
-
Clear design principle. "Compress what you see, not what you say" is well motivated. Keeping recent observations as raw text is justified by the finding that exact recitation at 16× compression is only about 15%.
-
Honest and detailed cost accounting. The paper reports encoder compute, prefix-cache reuse, call counts, and throughput. Negative results are not hidden.
Critical weaknesses
-
The method gives no net benefit over trivial baselines. On Qwen3, simple observation masking matches LOHA's resolve rate (12.2% vs. 12.1%) with less context (20.1K vs. 24.8K tokens per call) and higher throughput (64.7 vs. 63.4 instances/GPU-hour). Pruning baselines reach a higher resolve rate on both models. Total compute goes up rather than down (2.10 + 1.64 PF vs. 1.82 PF for the base; about 2.12 PF even with encoder caching), because the compressed agent makes more calls (60.9 vs. 37.9).
-
ACD does not preserve behavior, which is its stated goal. After ACD training, the model already loses performance with no compression at all (14.5% → 13.2% and 27.5% → 24.3% at K=∞), so the anchor does not prevent behavioral drift. The main positive results (the 32K limit and the 1.9× throughput) are measured mostly against this degraded model. Against the original base, the gains are small (63.4 vs. 58.1 instances/GPU-hour) or come from a single run on a 199-task subset.
Questions for authors
-
Is there any setting where LOHA beats both the uncompressed base and masking on resolve rate and total compute at the same time?
-
Does the 32K result (42 vs. 31 tasks against the base) hold up across multiple seeds?
3 comments
Thank you again for your review. As the discussion period closes on November 18, we wanted to check whether our response addresses your questions. The revised PDF reports measured encoder work (total 2.11 vs. 1.82 PF for the base; Section 3.4), the six-seed 32K results against the original base (Section 3.2; Appendix B.2, Table 5), pooled comparisons of the adapted agent with its base (Appendix B.1, Table 4), and the anchor separation with measured drift (Appendix B.4, Table 8). If these resolve your concerns, we would be grateful if you would consider updating your assessment.
Official Comment
Thanks for the new experiments. I maintain my rating.
- No setting beats both the base and masking on resolve rate and compute. At the full window LOHA still resolves fewer tasks (12.1 vs. 14.5; 21.8 vs. 27.5) at higher compute (2.11 vs. 1.82 PF), and resolved tasks per GPU-hour stay below both at every reported point.
- The 32K gain is marginal: +2.1 points, and the seed-level interval [-0.3, 4.5] covers zero.
- Removing the anchor does not change resolve rate (p=0.69, 0.54), so its value for task success is not shown.
Response to Reviewer 6ygG: Tight-Budget Advantage, Base-Relative Results, and Behavior Preservation
Thank you very much for your careful review, for recognizing the design principle and the transparent cost accounting, and for your clear and direct questions. We address them in turn; section and table numbers refer to the submission.
Q1 (a setting where LOHA beats both the base and masking on resolve rate and compute). We appreciate this pointed question. The closest is a tight context budget: under a 32K limit, LOHA resolves the most of all arms (34.3 per run vs. 30.2 for the base and 27.7/30.5 for the two masking variants; six seeds) while using less compute than either masking variant (decoder 0.74 vs. 0.90 and 0.95 PF per trajectory). Of these, only the base is cheaper (0.21 PF), because 42% of its trajectories are cut off at the cap. At the full window, two points change the trade-off:
Q2 (32K across seeds). Thank you for this question. All five arms now have six seeds (s1–s6) on the same 199 instances. LOHA has the highest mean (Q1): +2.1 points over the base (95% CI [0.1, 4.3]) and +3.4 over base + masking ([1.2, 5.7]). It resolves at least as many instances as the base in every seed and hits the 32K cap less than half as often (34.7 vs. 83.7 trajectories per run, fewer in all six seeds). Per seed, LOHA resolves 42, 33, 32, 35, 33, and 31 against 31, 25, 32, 31, 33, and 29 for the base.
W1 (comparison with simple baselines). As Section 3.1 notes, the pruning baselines that resolve more keep larger contexts (29–49K tokens per call, against 22–25K for LOHA), and Table 1 shows that masking with the same window makes as many calls as LOHA (60.6 vs. 60.9), so the extra calls come with removing older observation text rather than with latent encoding. We would also like to respectfully clarify that all reduction baselines, including masking, ran on the adapted model, so they isolate the context layout. Over 499 instances with 2–3 seeds per arm:
On the stronger agent, keeping the history in latent form is significantly better than deleting it, so the compressed observations are read and used. Masking on the untrained base is a different comparison, since the weights change too, but it is the relevant zero-training alternative. Under a 32K limit, LOHA resolves 3.4 points more than it (34.3 vs. 27.7, six seeds); on SWE-Master at the full window the two are comparable (54/53 vs. 54/50 on 199 instances). LOHA's distinguishing property is that older observations are retained in compressed form rather than discarded, within one model that also runs on plain text.
W2 (does ACD preserve behavior). Thank you for this question.
We hope these results clarify what LOHA + ACD provides and how its costs compare with the alternatives. Thank you again for your time and feedback; we would be happy to provide any further details.