- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Glasseek is an open recipe for training 4B and 9B deep-search agents. The data comes from a multi-agent pipeline that writes hard search questions. Training uses a GSPO variant built for rollouts where the context gets compressed. At evaluation time, a filter blocks the agent from looking up benchmark answers online. The authors also audit several existing agents and find that they do look up answers. The 9B model reports 61.6 on BrowseComp.
Critical strengths
I learned the most from the leakage study. In Appendix F, agents retrieve benchmark answers or previous solution traces from public web resources. Anyone who runs these benchmarks should see those examples. Appendix D's discussion of selective restarts is useful too, especially the distinction between single-episode capability and gains produced by giving resource-exhausted runs fresh attempts.
Critical weaknesses
The main problem is that the paper does not fully follow its own evaluation argument. Appendix D explains that selective restarts can increase the reported task-level score without improving single-episode reasoning ability, and recommends treating this as a protocol-level metric. Yet the headline BrowseComp number, 61.6, uses this restart protocol. The corresponding first-attempt accuracy is 54.5, which is below the 59.2 Gemini-3-Pro number used in the paper's comparison. Table 2 also states that many unmarked baseline scores are taken from their original reports and retain their original evaluation settings. This makes the broad headline comparison difficult to interpret. In particular, the abstract says the 9B model outperforms frontier proprietary agents across almost all deep-research benchmarks, which does not seem supported by Table 2 under a common protocol.
I also found the RL contribution hard to isolate. I could not find repeated training runs for the RL stage, the exact definition of the "+ Compaction" baseline is not sufficiently clear to me, and the two main ingredients of CM-GSPO are not separately ablated. The terminology also switches between turn-level and sub-trajectory importance ratios in different parts of the paper, and I think this distinction should be made more explicit.
One more result needs clarification. For BrowseComp, the CM-GSPO entry in Table 3 is exactly 61.6, the same as the full-result entry in Table 2. However, Section 5.3 says the BrowseComp ablation is evaluated on a 200-question subset and averaged over three independent runs. This could be coincidence, but the identical value makes me wonder whether one of the numbers comes from the full evaluation rather than the stated ablation protocol. The paper's own leakage rate is also left blank in Table 1.
Minor: the reproducibility statement says representative leakage trajectories are in Appendix E, while the paper itself places them in Appendix F.
Questions for authors
-
Can you put the headline model comparison under one consistent protocol, ideally the same harness with first-attempt results, and revise the abstract accordingly? This would affect my score more than anything else.
-
Do the Table 3 ablations use the restart protocol? Also, can you clarify why the BrowseComp CM-GSPO number is exactly the same as the full-set Table 2 result despite the stated 200-question, three-run ablation setup?
-
How often does Glasseeker itself attempt to access benchmark artifacts when the anti-leakage filter is disabled? Also, did you test the synthetic training questions for overlap or near-duplication with the evaluation sets?
3 comments
Q3. Access to benchmark artifacts and overlap with the training data
Access to benchmark artifacts with the filter disabled (Table 1). We ran an additional evaluation with the anti-leakage filter disabled. Glasseeker-9B attempted to access benchmark-related artifacts in 13.8% of evaluation trajectories. The 61.6% accuracy reported on the full BrowseComp test set was obtained with the leakage filter enabled and selective restarts allowed.
Overlap with English BrowseComp (Appendix G). We checked the released training questions for near-duplicates of English BrowseComp questions. After removing fixed prompt templates and deduplicating within each training split, we compared 4,359 SFT questions and 1,308 RL questions against all 1,266 English BrowseComp questions. We embedded the question texts with Qwen3-Embedding-4B, retrieved the nearest BrowseComp question for each training question by cosine similarity, and inspected the highest-scoring pairs. The maximum similarity was 0.7915 for SFT and 0.7851 for RL.
Appendix G shows the three closest SFT pairs and the two closest RL pairs. In the closest SFT pair, both questions use a supervisor as a clue to identify a thesis. The training question asks for the second supervisor listed in the archive, while the BrowseComp question asks for the author’s surname. In the closest RL pair, both questions concern early-1990s PC games. One asks for the developer and the other for the game title. These examples illustrate why we examined the entities, constraints, and requested answers alongside each similarity score. The scope of this audit is question-text overlap with English BrowseComp. Other evaluation sets and possible exposure during pretraining remain outside its scope.
RL baselines and component ablations
The revised Table 3 identifies the importance-ratio unit and whether each variant uses context compaction and length-based reweighting. A session is the part of a rollout between compaction boundaries. The configurations differ as follows.
- Vanilla GSPO computes one importance ratio per session and averages token losses within that session. With compaction disabled, a session spans the entire rollout.
- + Compaction keeps the same rule while enabling context compaction. Each sub-trajectory now has its own ratio and independently averaged loss. Summing these losses gives greater aggregate weight to rollouts split into more sub-trajectories.
- Our ReSum-GRPO implementation uses token-level ratios with compaction and broadcasts the rollout advantage to every segment. It applies no segment-length correction.
- + Compaction & Turn IS keeps compaction and the same sub-trajectory loss normalization as + Compaction, but computes one ratio per turn. Comparing these rows isolates the choice of ratio unit.
- CM-GSPO keeps the turn-level ratio and weights each sub-trajectory’s mean loss by its share of the rollout’s unmasked execution tokens, . These weights sum to one within each rollout, removing the extra weight introduced by splitting it into more segments. Comparing CM-GSPO with + Compaction & Turn IS isolates this reweighting.
Equations (3) and (4) use to index the turn that supplies the ratio. Equation (5) defines the sub-trajectory weight, and Equation (6) combines the losses within each rollout.
On BrowseComp-200, accuracy rose from 55.0% with + Compaction to 57.0% with + Compaction & Turn IS, a gain of 2.0 percentage points from switching to turn-level ratios. Adding rollout-level reweighting (CM-GSPO) raised accuracy to 60.5%, a further 3.5 percentage points and a total gain of 5.5 percentage points over + Compaction.
Reproducibility statement
We corrected the reference to Appendix F, which contains the representative leakage trajectories and their event indices. Appendix E reports the topic distribution of the SFT data.
Dear Reviewer sQvB,
Thank you again for your careful and constructive review. As the discussion period is coming to a close, we wanted to kindly check whether our response has addressed your concerns, in particular:
- Q1 (common protocol): We added a first-attempt comparison in which all checkpoints use the same harness, questions and limits (Table 5), and revised the abstract to rest on this comparison and to report both first-attempt (54.5%) and restart (61.6%) BrowseComp results.
- Q2 (Table 3 protocol): We clarified that the ablations use selective restarts on a single run over the fixed 200-question subset, and corrected the CM-GSPO entry to 60.5% (121/200).
- Q3 (leakage and overlap): With the filter disabled, Glasseeker-9B attempts to access benchmark artifacts in 13.8% of trajectories; we also added a near-duplicate audit of the training questions against BrowseComp (Appendix G).
We would be happy to provide any further clarification. If our responses have resolved your concerns, we would be grateful if you would consider updating your evaluation.
Thank you for highlighting the leakage examples in Appendix F and the distinction in Appendix D between first-attempt accuracy and task completion with selective restarts. We agree that the headline comparison should make this distinction explicit.
Q1. Headline comparison and first-attempt evaluation
On the full BrowseComp test set, Glasseeker-9B achieves 54.5% accuracy on the first attempt and 61.6% with selective restarts. Under the protocol in Appendix D, an attempt has a budget of 500 interaction turns. A question receives a fresh attempt only if the agent exhausts its budget before producing a final answer, with at most five attempts in total. The first final answer ends the evaluation, regardless of correctness. The 7.1 percentage point increase measures the gain from this restart policy.
Gemini-3-Pro’s published BrowseComp score of 59.2% exceeds our first-attempt score of 54.5%. Table 2 identifies the baselines we evaluated in our harness with a dagger. The Gemini score comes from its original report. We agree that the original abstract overstated what comparisons across different evaluation protocols could establish. The revised abstract bases its comparison on Table 5, where all checkpoints used the same harness, answered the same questions, and received one attempt per question with identical turn and time limits. Under this protocol, Glasseeker-9B outperforms both open-source models with fewer than 9B parameters on all five benchmarks. The abstract also reports both results on the full BrowseComp test set: 54.5% accuracy on the first attempt and 61.6% with selective restarts.
Comparison under a common protocol. We added a first-attempt comparison in Section 5.2 and report the results in Table 5 of Appendix D. We evaluated all five checkpoints on the same questions using the same Hermes-agent harness, with identical turn and time limits and one attempt per question. Appendix D gives the remaining evaluation settings. The table reports first-attempt accuracy (%) with the number of questions in parentheses.
We could not include Gemini-3-Pro in Table 5 because we did not have access to its API during this evaluation.
Apodex scored poorly in part because it requested tools in a format our harness could not reliably read. MiroThinker scored 30.5% on our 200 BrowseComp questions and 31.1% on the original full set, and neither evaluation use the restart protocal.
Glasseeker-9B also scores above Apodex-1.0-4B-SFT on all five benchmarks. Its accuracy is close to Claude-Opus-4.5 on BrowseComp-200 (54.5% versus 56.5%) and Seal-0 (40.5% versus 41.4%). On GAIA-Text, Glasseeker-9B scores 72.8%, compared with 68.0% for Claude-Opus-4.5 and 79.6% for DeepSeek-V4-Pro.
Q2. Restart policy for Table 3 and the earlier 61.6% entry
Yes. All BrowseComp ablations in Table 3 used selective restarts. Each attempt had a limit of 500 interaction turns and 3,600 seconds. If the agent reached either limit before producing a final answer, it could retry the question, up to five attempts including the first. Evaluation stopped as soon as the agent produced a final answer.
Correction to the submitted Section 5.3. We incorrectly stated that Table 3 reported ablation results averaged over three independent runs. We have corrected this reporting error in Section 5.3. Each BrowseComp score in Table 3 comes from one evaluation run per checkpoint on the same fixed subset of 200 questions. Selective restarts took place within that run.
Table 2 reports 61.6% accuracy on the full BrowseComp test set. The identical 61.6% entry for CM-GSPO in the submitted Table 3 was a typographical error. The corrected Table 3 entry is 60.5%. In the single subset run, CM-GSPO solved 121 of 200 questions, compared with 111 of 200 for the SFT baseline (55.5%), a gain of 5.0 percentage points.