- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes POSEIDON as a pipeline for generating training environments from code repositories. The authors also constructed PEARL, a collection of such environments. Experiments showed that training on PEARL improved Qwen 3.5 models performance on six scientific-agent benchmarks.
Critical strengths
- POSEIDON extracts 10 times more training environments than a prior approach, making more efficient use of existing code repositories.
- PEARL, as the largest collection, contributes to the community and the scientific agent field.
- Manual review ensures task quality, although the paper only sampled 600 tasks for review.
Critical weaknesses
- The code link provided https://anonymous.4open.science/status/POSEIDON-89D7 is incorrect and contains no content. It should be https://anonymous.4open.science/r/POSEIDON-89D7. Moreover, the source commit modification date is October 8, 2026, many days after the submission deadline. This is not good practice and unfair to other timely submissions.
- Distribution of the collected scientific repositories across the disciplines are very unbalanced, with biology accounting for too high a percentage at 47%. Is a more balanced distribution possible? The authors mention oversampled the biology repository because of its active development, but 47% is still too high.
- The manual review acceptance rate is 95.5% for 600 tasks sampled. Ideally, all environments should undergo manual review. Since the acceptance rate isn't 100%, it means some aspects are problematic. However, I understand this would be very costly. Another approach might be to summarize the remaining 0.5% and identify the problems, then extract common issues and perform model-assisted filtering on all environments.
Questions for authors
Please address the weaknesses above. Figure 3: The title is too short, without information for components a, b, and c.
7 comments
Response to Reviewer c2U5 (Part2/3)
Analysis of Human Audit Findings and Targeted Filtering (W3)
Thank you for this constructive suggestion. We reviewed the cases flagged during the human audit and found that most involved isolated presentation issues that did not affect task solving or evaluation, such as inconsistent capitalization, redundant instructions, and overly detailed background descriptions. Two tasks, however, shared a terminology issue: the scientific metric names did not precisely match the computations specified in the instructions. A representative example is provided below.
Task
You are provided with data/validation_coords.pt, a dictionary containing
three NumPy arrays:
- p1: predicted coordinates of shape
(num_samples, num_residues, num_atoms, 3)
- p2: true/experimental coordinates of the same shape
- mask: a boolean array of shape (num_samples, num_residues), where
1 indicates a valid residue and 0 an invalid residue
Your goal is to evaluate the accuracy of the predicted protein coordinates
against the experimental structures using Global Distance Test (GDT)
scores. Do not perform any structural superposition or alignment; use
the coordinates exactly as provided. Use only the C-alpha atom of each
residue (atom index 0 along the atom axis).
For each sample, compute:
- The Euclidean distance between the predicted and true C-alpha
coordinates for each residue.
- Restrict evaluation to residues marked valid by mask; only valid
residues contribute to the numerator and denominator.
- For a given distance cutoff c, the fractional score is:
(# valid residues with distance <= c) / (# valid residues).
- GDT_TS is the average of these fractional scores over cutoffs
[1, 2, 4, 8] Å.
- GDT_HA is the average of these fractional scores over cutoffs
[0.5, 1, 2, 4] Å.
Compute these per-sample scores for the whole validation set, then
aggregate them across samples. Report four aggregate values:
mean GDT_TS, median GDT_TS, mean GDT_HA, and median GDT_HA.
Issue
The task contains a terminology mismatch. Standard GDT involves searching
for structural superpositions to evaluate the fraction of corresponding
residues within specified distance thresholds. This task instead computes
distances directly in the supplied coordinate frames, making the scores
sensitive to overall rotation and translation. Although the requested
computation is explicitly specified, calling these quantities GDT_TS and
GDT_HA without qualification is potentially misleading. More precise names
would be "raw-frame distance-threshold scores" or "GDT-like scores without
superposition."
We translated this recurring issue into a targeted model-assisted screening criterion and applied the following prompt across the full collection:
You are a scientific terminology reviewer.
Check whether the scientific names used in the following task accurately
describe the computations defined in the task itself. Identify names that
are misleading because the stated computation omits or changes an essential
part of their conventional definitions.
Return only valid JSON:
{
"has_misleading_names": true,
"issues": [
{
"name": "",
"mismatch": "",
"suggested_name": ""
}
]
}
If no misleading names are found, set "has_misleading_names" to false
and "issues" to [].
Task:
{prompt}
This screening flagged 127 environments, approximately 0.13% of the full collection. We removed these environments and their associated training trajectories.
Human review before large-scale generation. Human auditing was also conducted during pilot generation, before we scaled up production. These early reviews identified more substantial recurring problems, including inconsistent data-path prefixes (data/ versus data/data/), graders incorrectly accepting invalid or non-answer outputs such as NaN, None, empty lists, and empty dictionaries, and tasks unrelated to scientific reasoning, such as scraping paper-author information or troubleshooting GPU memory capacity. We addressed these issues before proceeding with large-scale generation.
Thus, human review served both to improve the generation pipeline before scaling and to identify residual issues for targeted filtering afterward.
Response to Reviewer c2U5 (Part3/3)
Expanded Caption for Figure 3 (Q1)
We have expanded the caption of Figure 3 to provide clearer descriptions of components a, b, and c.
The detailed caption now is:
Figure 3: Overview of POSEIDON. (a) Repo Discovery: expands subject keywords with an LLM, searches GitHub for repositories meeting our requirements on stars, data, and paper availability, and builds an executable sandbox for each. (b) Task Synthesis: groups a repository’s code files by scientific workflow and derives four task types from them: Reproduction, Subflow, Completion, and Correction. (c) Verification: a generator–validator loop with recursive feedback ensures environment reliability and scientific validity.
Response to Reviewer c2U5 (Part1/3)
Correction of the Code Link and Clarification of Release Updates (W1)
Thank you for pointing out the incorrect link and raising your concern about the timing of the code update. We sincerely apologize for providing the repository status page instead of the code-browsing page. The correct link is https://anonymous.4open.science/r/POSEIDON-89D7. We have corrected the code link on the first page of the revised manuscript.
The October 8 update was intended to address omissions and outdated components in the public code release. We updated the
qcdirectory, which contains the quality-control workflow and the Docker image build-and-push workflow. We also removed outdated code for two task types explored during early development, retaining the four task types described in the paper.These updates were intended to improve the completeness and consistency of the public release. Nevertheless, we recognize that we should have completed this preparation earlier. We sincerely apologize for the delayed update and the confusion and additional review effort it caused. Thank you for bringing these issues to our attention.
Training with a More Balanced Disciplinary Distribution (W2)
Thank you for highlighting the disciplinary imbalance. To investigate whether a more balanced training distribution can retain the benefits of our environments, we constructed a smaller SFT dataset by downsampling biology and pharmacology trajectories from the original training collection. The resulting dataset contains 36,721 trajectories, approximately 57% of the original 64,477 trajectories, with no single discipline accounting for more than 22.2%.
We then fine-tuned Qwen3.5-35B-A3B on this rebalanced dataset. The results are shown below.
Despite using approximately 43% fewer trajectories, the rebalanced dataset improves the average score from 41.6 to 45.7, with gains on six of the eight evaluation metrics and comparable performance on the remaining two. Although the rebalanced subset achieves a lower average score than the full dataset, it retains clear overall gains over the untuned backbone despite using substantially fewer trajectories. The performance difference may reflect the combined effects of reduced data quantity and the adjusted disciplinary composition.
These results demonstrate that a more balanced training mixture is feasible and retains training value. The benefits of our environments are therefore not restricted to the original disciplinary distribution.
We have added this experiment to Appendix H, with the rebalanced data distribution in Table 10 and the evaluation results in Table 11.