- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
ACM uses manage_context and query_memory to compress working context while preserving raw messages in external memory for later retrieval. Its teacher-guided data generation pipeline uses dual constraints, Injection and Refinement, to teach when to use context management tools and when not to call them. Experiments use Qwen3.5-9B on BrowseComp-Plus, DeepSearchQA, and SWE-Bench Verified. On BrowseComp-Plus, Table 2 reports a Pass@1 increase from 57.0 to 72.7 percent over ReAct, with lower peak token usage and more tool calls.
Critical strengths
-
The context management tools address a practical limitation of summarization: a discarded detail can be needed later. Preserving raw messages in external memory allows the agent to query that evidence through query_memory. The reported results also separate several stages of the system. On BrowseComp-Plus, adding the ACM tools improves accuracy by 6.5 percentage points over ReAct, and ACM training adds a further 9.2 points. Gains after training are smaller on DeepSearchQA and SWE-Bench Verified, at 2.0 and 1.6 points over the base ACM agent. This is more informative than reporting only a comparison between the final system and ReAct. The appendix provides tool schemas and teacher prompts that help explain the implementation.
-
The distillation ablation is useful and shows that the effect depends on the task. In Table 3, GPT-5.5 distillation reaches 62.3, 38.1, and 64.2 percent on the three benchmarks, whereas ACM training reaches 72.7, 42.5, and 63.0 percent. ACM training is stronger on the search tasks, while GPT-5.5 distillation is stronger on coding. Combining them gives the highest reported values in that table, at 78.5, 43.0, and 65.6 percent. This suggests that the teacher-annotated data provides a different training signal from GPT-5.5 distillation, although it does not isolate which changes within those trajectories produce the gain.
Critical weaknesses
-
Refinement can replace context management calls with search queries or final answers, so the training data changes search and answer behavior as well as context management. Table 3 compares this combined supervision with GPT-5.5 distillation, but does not isolate which changes produce the gain. A control retaining the same search and answer corrections while omitting context management supervision would help determine how much of the improvement comes from learning when to manage context.
-
The SWE-Bench Verified evaluation needs clarification. The AI use statement says that final answers in all evaluations are graded by an LLM, but the grading description covers only the search tasks and does not specify whether repository tests were used for coding. Tables 2 and 3 also report 72.7 percent Pass@1 on BrowseComp-Plus, while Figure 5 gives a four trial average of 76.2 percent. These values may come from different runs, but the relationship is not explained. The scoring procedure and aggregation rules should be stated explicitly for each reported result.
Questions for authors
-
What evidence separates context management from improved search and answer supervision? Is there a comparison using the same targeted corrections and search instructions while omitting context management actions, or an ablation that removes query_memory while retaining manage_context?
-
Which runs and aggregation rules produced the 72.7 percent value in Tables 2 and 3 and the 76.2 percent average in Figure 5? Can you report paired uncertainty for the gains over the base ACM agent, particularly the smaller gains on the two out-of-domain benchmarks?
4 comments
Table R3. Strategy-prompt control.
| Agent | Context-management tools | Strategy instructions | BrowseComp-Plus Pass@1 | DeepSearchQA Pass@1 |
|---|---|---|---|---|
| ReAct | ✗ | ✗ | 57.0 | 36.7 |
| ReAct + strategy prompt | ✗ | ✓ | 59.3 | 37.9 |
| ACM Base | ✓ | ✓ | 63.5 | 40.5 |
The instructions improve accuracy somewhat, but the context-management tools contribute the larger share of the gain.
W2 / Q2: Scoring, aggregation and uncertainty
SWE-Bench Verified scoring. We thank the reviewer for pointing out this oversight. SWE-Bench Verified is graded entirely by execution, not by an LLM. We follow the official SWE-bench harness and host each instance's environment on Modal to run its tests; an instance counts as resolved only if all FAIL_TO_PASS and PASS_TO_PASS tests pass. The LLM judge in App. B.6 is used only for BrowseComp-Plus and DeepSearchQA. We will correct the AI-use statement.
Aggregation, and 72.7% vs. 76.2%. All Pass@1 values are mean accuracy over 4 independent trials on the same test set (150 held-out questions for BrowseComp-Plus). Pass@4 counts a question as solved if any of the 4 trials is correct. The 72.7% in Tables 2–3 is the 4-trial mean of our released checkpoint. The 76.2% in Figure 5 comes from a different checkpoint: we first trained with LLaMA-Factory and later moved to a custom Transformers trainer, the version in our released code, and Figure 5 was drawn by mistake from the earlier checkpoint. We apologize for the inconsistency. Figure 5 will be regenerated from the released checkpoint, which we share through a newly created anonymous Hugging Face account: https://huggingface.co/anonymous-openreview/acm-post-trained-qwen3.5-9b.
Paired uncertainty. Table R4 reports paired 95% confidence intervals for every gain.
Table R4. Paired uncertainty of the gains.
| Comparison | Benchmark | Δ Pass@1 | 95% CI |
|---|---|---|---|
| ACM Base − ReAct | BrowseComp-Plus | +6.5 | [5.2, 7.6] |
| ACM Base − ReAct | DeepSearchQA | +3.8 | [2.1, 5.3] |
| ACM Base − ReAct | SWE-Bench Verified | +3.4 | [2.5, 3.9] |
| ACM-Post-Trained − ACM Base | BrowseComp-Plus | +9.2 | [7.3, 11.4] |
| ACM-Post-Trained − ACM Base | DeepSearchQA | +2.0 | [0.9, 3.2] |
| ACM-Post-Trained − ACM Base | SWE-Bench Verified | +1.6 | [0.8, 2.1] |
The in-domain training gain on BrowseComp-Plus is large. The out-of-domain training gains are smaller but statistically reliable, even though training uses BrowseComp-Plus only. Table R1 above shows where this transfer comes from: without context-management supervision, it largely disappears. We will add Table R4 and the aggregation rules to the revised paper.
I appreciate the authors’ response and their answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.
Thank you for your time and effort in reviewing our paper! If you have any other questions or concerns, please feel free to let us know—we’d be happy to answer them!
We thank the reviewer for the detailed review. We address the weaknesses and questions with new experiments, including the suggested control.
W1 / Q1: Separating context management from search and answer supervision
The suggested control. The teacher sees the gold answer and evidence when annotating, but Injection only inserts
manage_context/query_memorycalls, so Refinement is the only path through which gold evidence can shape the training data.Table R1 implements the control the reviewer suggested: the masked variant retains exactly the same targeted search and answer corrections and omits only the context-management supervision. We also train on Injection-only and Refinement-only data, generating additional demonstrations with the same pipeline so that every variant uses the same number of demonstrations. All trained variants use 1,880 teacher-annotated demonstrations and share the student, teacher, NLL objective and hyperparameters. The masked variant trains on exactly our trajectories, with the loss zeroed on every assistant turn that calls
manage_contextorquery_memory, including its reasoning.Table R1. Training-data ablation.
Removing
query_memorywhile retainingmanage_context. Compression is still decided by the agent but becomes lossy.Table R2. Effect of removing
query_memory.query_memoryRemoving recall lowers BrowseComp-Plus accuracy by 1.5 points for ACM Base and 2.0 points for ACM-Post-Trained, with smaller drops on DeepSearchQA. Most of ACM's gain comes from agent-initiated compression, and
query_memorycomplements it. The agent recalls only about once per episode (Fig. 4), yet these few recalls are worth 1.5–2.0 points. Together, the two tools give the agent a short-term working context and a long-term store it can return to, a combination we see as promising for longer tasks.The search instructions. Compared with the ReAct prompt, the ACM prompt adds the following strategy instructions:
Table R3 adds exactly these instructions to ReAct, without the context-management tools.